[08:25:53] <Damianz>	 :D
[08:25:56] <Damianz>	 http://labs.wikimedia.beta.wmflabs.org/w/index.php?title=Help < is what I mean, should probably go on proper labs.
[08:29:29] <petan|wk>	 ok
[08:29:36] <petan|wk>	 I will make a doc on meta:WM-Bot
[08:29:42] <petan|wk>	 because that's a home of wm-bot
[08:29:49] <petan|wk>	 @help
[08:29:49] <wm-bot>	 Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.1.4 source code licensed under GPL and located in wikimedia svn
[08:31:04] <petan|wk>	 how is it possible that mysql is broken damn
[08:31:14] <petan|wk>	 Ryan_Lane: why fs was corrupted?
[08:31:24] <petan|wk>	 I should run fs checks on all instances then
[08:31:30] <Ryan_Lane>	 well, the fs wasn't
[08:31:41] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on deployment-web deployment-web output: All packages OK
[08:31:41] <petan|wk>	 how could the whole datafile could disappear?
[08:31:46] <Ryan_Lane>	 project storage was unaffected
[08:31:51] <Ryan_Lane>	 I have no idea
[08:31:51] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on deployment-web deployment-web output: OK - load average: 0.18, 0.14, 0.06
[08:31:51] <petan|wk>	 that's weird
[08:31:51] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on deployment-web deployment-web output: USERS OK - 0 users currently logged in
[08:31:51] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on deployment-web deployment-web output: DISK OK
[08:32:04] <petan|wk>	 maybe more databases were broken
[08:32:22] <petan|wk>	 Damianz: root password is puppet ;)
[08:32:26] <Damianz>	 Tbf innodb doesn't like being turned on and off randomly and hard and having the fs yanked from under it :P
[08:32:28] <petan|wk>	 you could search logs for that
[08:32:39] <petan|wk>	 btw I configured sql so sudo mysql works
[08:32:48] <Damianz>	 petan|wk: Oooh, I could have - .my.cnf would be nice though :D
[08:32:56] <petan|wk>	 that's what should be there
[08:33:29] <petan|wk>	 Damianz: question is if more tables are broken or not
[08:33:36] <petan|wk>	 if there is problem, we should restore it asap
[08:33:41] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on deployment-web deployment-web output: OK: 82% free memory
[08:33:41] <labs-nagios-wm>	 RECOVERY SSH is now: OK on deployment-web deployment-web output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[08:33:41] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on deployment-web deployment-web output: PROCS OK: 107 processes
[08:34:08] <Damianz>	 Mhm, I couldn't get past what was moaning in the log so I have no idea -- suspect some others may be, not a simple way to tell unless the wiki's come up and the log starts getting spammed with errors.
[08:34:48] <petan|wk>	 Damianz: wiki is up
[08:34:52] <petan|wk>	 it's already fixed
[08:36:41] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on deployment-dbdump deployment-dbdump output: All packages OK
[08:36:51] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on deployment-dbdump deployment-dbdump output: OK - load average: 0.23, 0.14, 0.05
[08:36:51] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on deployment-dbdump deployment-dbdump output: USERS OK - 0 users currently logged in
[08:36:51] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on deployment-dbdump deployment-dbdump output: DISK OK
[08:38:41] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on deployment-dbdump deployment-dbdump output: OK: 94% free memory
[08:38:41] <labs-nagios-wm>	 RECOVERY SSH is now: OK on deployment-dbdump deployment-dbdump output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[08:38:41] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on deployment-dbdump deployment-dbdump output: PROCS OK: 98 processes
[08:41:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[08:41:51] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[08:41:51] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[08:41:51] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[08:41:51] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[08:51:51] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[08:51:51] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[08:51:52] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[08:51:52] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[08:51:52] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[09:11:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[09:11:52] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[09:11:52] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[09:11:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[09:11:52] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[09:21:51] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[09:21:51] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[09:21:51] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[09:21:51] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[09:21:51] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[09:41:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[09:41:51] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[09:41:52] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[09:41:52] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[09:41:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[09:51:51] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[09:51:51] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[09:51:52] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[09:51:52] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[09:51:52] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[09:52:01] <labs-nagios-wm>	 RECOVERY host: dumps-4 is UP address: dumps-4 PING OK - Packet loss = 0%, RTA = 1.39 ms
[09:58:02] <Hydriz>	 !log incubator Instance incubator-web is going very crazy right now, all web services for incubator are down
[09:58:03] <labs-morebots>	 Logged the message, Master
[10:08:41] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on incubator-web incubator-web output: DISK OK
[10:11:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[10:11:51] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[10:11:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[10:11:52] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[10:21:51] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[10:21:51] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[10:21:51] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[10:21:51] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[10:21:51] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[10:41:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[10:41:51] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[10:41:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[10:41:52] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[10:51:51] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[10:51:51] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[10:51:52] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[10:51:52] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[10:51:52] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[10:51:58] <Hydriz>	 goddamn...
[11:11:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[11:11:51] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[11:11:51] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[11:11:51] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[11:21:54] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[11:21:55] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[11:21:55] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[11:21:55] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[11:21:55] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[11:41:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[11:41:51] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[11:41:51] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[11:41:51] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[11:52:01] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[11:53:41] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[11:53:41] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[11:53:41] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[11:53:41] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[12:05:35] <petan|wk>	 !nagios
[12:05:35] <wm-bot>	 http://nagios.wmflabs.org/nagios3
[12:11:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[12:11:52] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[12:11:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[12:11:52] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[12:22:01] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[12:23:41] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[12:23:41] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[12:23:42] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[12:23:42] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[12:41:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[12:41:51] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[12:41:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[12:41:52] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[12:52:01] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[12:53:41] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[12:53:41] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[12:53:42] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[12:53:42] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[12:57:59] <petan|wk>	 mutante, ssmollett can you reboot all these instances reported by nagios?
[12:58:04] <IWorld>	 wm-bot is here :)
[12:58:12] <petan|wk>	 !ping
[12:58:12] <wm-bot>	 pong
[12:58:27] <Damianz>	 !pong
[12:58:33] * Damianz  kicks wm-bot

[12:58:38] <IWorld>	 Yesterday I have prepared wm-bot-helping
[12:58:39] <petan|wk>	 !pong is don't play with me
[12:58:39] <wm-bot>	 Key was added!
[12:58:53] <petan|wk>	 :o
[12:59:24] <IWorld>	 With some functions like rss, flood protection and more ;)
[12:59:29] <petan|wk>	 ah
[12:59:44] <petan|wk>	 you mean it's a feature request or something like that?
[12:59:52] <IWorld>	 no
[12:59:53] <petan|wk>	 I think I could implement that
[12:59:57] <petan|wk>	 hm
[12:59:59] <petan|wk>	 what you mean then
[13:00:56] <IWorld>	 see ##iworld
[13:01:12] <petan|wk>	 what am I going to see
[13:01:18] <petan|wk>	 anyway if there is idea
[13:01:20] <Damianz>	 ascii porn
[13:01:21] <petan|wk>	 to improve this bot
[13:01:26] <petan|wk>	 just do that
[13:01:27] <petan|wk>	 it's in svn
[13:02:16] <IWorld>	 I haven't got SVN access and I can't write in C(++|#)
[13:02:21] <petan|wk>	 we need more operation engineers living in europe
[13:02:31] <petan|wk>	 IWorld: ok, I will do it then
[13:02:32] <IWorld>	 Why?
[13:02:40] <petan|wk>	 IWorld: because I want that
[13:02:51] <IWorld>	 ah
[13:02:55] <petan|wk>	 and everybody knowz that I rule wikimedia
[13:03:42] <IWorld>	 You are the "Wikimedia community dev" (--> G+)
[13:03:56] <Damianz>	 petan|wk: ryan is a europeanperson really ;)
[13:04:06] <Hydriz>	 heh
[13:04:19] <IWorld>	 I'm from europe.
[13:04:40] <Damianz>	 Tbh fuck having more european ops people, I'd rather live out in Cali :P
[13:04:41] <petan|wk>	 IWorld: but ur not operation guy
[13:04:49] <petan|wk>	 heh
[13:04:50] <IWorld>	 yes...
[13:04:51] <petan|wk>	 true
[13:05:11] <petan|wk>	 is there any plan to move all devs to california rather than hire european ops?
[13:05:24] <petan|wk>	 because I would
[13:05:26] <petan|wk>	 if I could
[13:05:27] <petan|wk>	 :D
[13:05:31] <petan|wk>	 move
[13:05:48] <IWorld>	 @tell IWorld list
[13:05:56] <IWorld>	 oh
[13:05:59] <Damianz>	 lol
[13:06:01] <petan|wk>	 .
[13:06:09] <Damianz>	 That would be win, defeats the point of 24/7 coverage
[13:06:10] <IWorld>	 it runs (on query) ;)
[13:06:22] <petan|wk>	 it's not so fast
[13:06:29] <petan|wk>	 python is really slow I see
[13:06:29] <petan|wk>	 :D
[13:06:38] <IWorld>	 @tell petan|wk test
[13:06:43] <petan|wk>	 apart of that it consumes a lot of memoery
[13:06:51] <petan|wk>	 c# <3
[13:07:05] <IWorld>	 Python <3
[13:07:06] <petan|wk>	 my bot has 100kb
[13:07:13] <petan|wk>	 and eats less than 2mb of ram
[13:07:36] <IWorld>	 where I can get the ram of the a process?
[13:07:56] <petan|wk>	 ps aux | grep 'process name' | grep -v grep
[13:07:56] <petan|wk>	 :P
[13:08:14] <petan|wk>	 it doesn't work if your process is "grapefruit"
[13:08:16] <petan|wk>	 :D
[13:08:17] <IWorld>	 petan|wk: do you use linux as main os?
[13:08:24] <petan|wk>	 grepfruit I meant
[13:08:25] <petan|wk>	 :P
[13:08:41] <petan|wk>	 IWorld: guess
[13:08:59] <IWorld>	 hmm....
[13:09:02] <IWorld>	 :D
[13:10:18] <petan|wk>	 !pong
[13:10:18] <wm-bot>	 don't play with me
[13:10:18] <wm-bot-helping>	 Sorry, I don' know!
[13:10:23] <petan|wk>	 !log
[13:10:23] <wm-bot-helping>	 petan|wk: Error: You don't have the owner capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
[13:10:29] <petan|wk>	 this is problem
[13:10:30] <petan|wk>	 ^
[13:10:36] <petan|wk>	 !log deployment-prep
[13:10:37] <labs-morebots>	 Message missing. Nothing logged.
[13:10:49] <Hydriz>	 lulz
[13:11:14] <petan|wk>	 !log
[13:11:15] <IWorld>	 Ctrl-C at console... I want to copy the text :P
[13:11:21] <wm-bot-helping>	 petan|wk: Error: You don't have the owner capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
[13:11:26] <IWorld>	 @log
[13:11:26] <wm-bot-helping>	 IWorld: An error has occurred and has been logged. Please contact this bot's administrator for more information.
[13:11:29] <petan|wk>	 IWorld: can you do something about that
[13:11:34] <petan|wk>	 because we need !log here
[13:11:34] <IWorld>	 lol
[13:11:41] <IWorld>	 !logbook
[13:11:41] <wm-bot-helping>	 Dunno, IWorld.
[13:11:44] <IWorld>	 aah
[13:11:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[13:11:52] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[13:11:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[13:11:52] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[13:11:53] <IWorld>	 @list
[13:11:53] <wm-bot-helping>	 IWorld: Admin, AttackProtector, BadWords, Channel, ChannelLogger, ChannelStats, Config, Infobot, Internet, Limiter, Math, Misc, News, Owner, Plugin, RSS, Scheduler, Seen, ShrinkUrl, Status, String, Time, Topic, URL, User, Utilities, and Web
[13:11:58] <IWorld>	 @list log
[13:11:58] <wm-bot-helping>	 IWorld: Error: 'log' is not a valid plugin.
[13:12:04] <IWorld>	 mh
[13:12:12] <IWorld>	 @list ChannelLogger
[13:12:12] <wm-bot-helping>	 IWorld: That plugin exists, but has no commands.  This probably means that it has some configuration variables that can be changed in order to modify its behavior.  Try "config list supybot.plugins.ChannelLogger" to see what configuration variables it has.
[13:12:25] <IWorld>	 .ChannelStats
[13:12:25] <wm-bot-helping>	 IWorld: On #wikimedia-labs there have been 292 messages, containing 8986 characters, 1615 words, 22 smileys, and 6 frowns; 6 of those messages were ACTIONs.  There have been 31 joins, 5 parts, 17 quits, 0 kicks, 0 mode changes, and 1 topic change.
[13:13:06] <petan|wk>	 !log test
[13:13:06] <wm-bot-helping>	 petan|wk: Error: You don't have the owner capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
[13:13:06] <labs-morebots>	 Message missing. Nothing logged.
[13:13:08] <petan|wk>	 still
[13:13:10] <petan|wk>	 :o
[13:13:23] <petan|wk>	 it will be annoying :P
[13:13:57] <IWorld>	 !logbook is on http://bots.wmflabs.org/~petrb/logs/#wikimedia-labs
[13:13:57] <wm-bot>	 Key was added!
[13:13:57] <wm-bot-helping>	 Got it.
[13:14:02] <IWorld>	 !logbook
[13:14:02] <wm-bot>	 on http://bots.wmflabs.org/~petrb/logs/#wikimedia-labs
[13:14:02] <wm-bot-helping>	 I guess logbook is on http://bots.wmflabs.org/~petrb/logs/#wikimedia-labs, IWorld
[13:14:12] <petan|wk>	 my bot is faster :)
[13:14:22] <IWorld>	 my bot is faster
[13:14:27] <IWorld>	 <wm-bot-helping> I guess logbook is on http://bots.wmflabs.org/~petrb/logs/#wikimedia-labs, IWorld
[13:14:27] <IWorld>	 <wm-bot> on http://bots.wmflabs.org/~petrb/logs/#wikimedia-labs
[13:14:38] <petan|wk>	 !logbook del
[13:14:38] <wm-bot-helping>	 Dunno.
[13:14:38] <wm-bot>	 Successfully removed logbook
[13:14:50] <IWorld>	 thats the problem...
[13:14:52] <IWorld>	 !logboo
[13:14:53] <wm-bot-helping>	 I have no idea!
[13:14:55] <IWorld>	 !logbook
[13:14:56] <wm-bot-helping>	 Somebody said logbook is on http://bots.wmflabs.org/~petrb/logs/#wikimedia-labs, IWorld
[13:15:34] <IWorld>	 argh
[13:15:48] <petan|wk>	 @commands
[13:15:48] <wm-bot>	 Commands: channellist, trusted, trustadd, trustdel, infobot-off, refresh, infobot-on, drop, whoami, add, reload, help, RC-, recentchanges-on, recentchanges-off, logon, logoff, recentchanges-, recentchanges+, RC+
[13:15:48] <wm-bot-helping>	 petan|wk: add, alert, announce, announce add, announce list, announce remove, any, apply, apropos, at, author, ban add, ban list, ban remove, base, calc, capabilities, capability add, capability list, capability remove, capability set, capability setdefault, capability unset, change, changename, channel, channels, channelstats, chr, cmd, commands, config, contributors, convert, cpu, ctime, cycle, (3 more messages)
[13:16:25] <IWorld>	 aah
[13:16:29] <IWorld>	 @forget logbook
[13:16:30] <wm-bot-helping>	 10-4!
[13:16:34] <IWorld>	 ?
[13:16:39] <IWorld>	 @logbook
[13:16:40] <wm-bot-helping>	 Wish I knew.
[13:16:40] <petan|wk>	 @forget
[13:16:41] <wm-bot-helping>	 Wish I knew, petan|wk.
[13:16:51] <petan|wk>	 @remove
[13:16:51] <wm-bot-helping>	 petan|wk: Error: The command "remove" is available in the BadWords, News, RSS, Scheduler, and Topic plugins.  Please specify the plugin whose command you wish to call by using its name as a command before "remove".
[13:17:27] <petan|wk>	 .
[13:17:31] <petan|wk>	 wm-bot-helping: :o
[13:17:32] <wm-bot-helping>	 I have no idea.
[13:17:37] <petan|wk>	 it talks too much
[13:17:58] <IWorld>	 @unload Infobot
[13:17:58] <wm-bot-helping>	 IWorld: The operation succeeded.
[13:17:58] <petan|wk>	 !bottalk is wm-bot-helping: hi :)
[13:17:58] <wm-bot>	 Key was added!
[13:18:00] <wm-bot-helping>	 petan|wk: Error: "bottalk" is not a valid command.
[13:18:03] <petan|wk>	 !bottalk
[13:18:03] <wm-bot>	 wm-bot-helping: hi :)
[13:18:03] <wm-bot-helping>	 petan|wk: Error: "bottalk" is not a valid command.
[13:18:05] <wm-bot-helping>	 wm-bot: Error: "hi" is not a valid command.
[13:18:07] <petan|wk>	 !bottalk
[13:18:08] <wm-bot>	 wm-bot-helping: hi :)
[13:18:08] <wm-bot-helping>	 petan|wk: Error: "bottalk" is not a valid command.
[13:18:09] <wm-bot-helping>	 wm-bot: Error: "hi" is not a valid command.
[13:18:18] <IWorld>	 @load Infobot
[13:18:18] <wm-bot-helping>	 IWorld: The operation succeeded.
[13:18:31] <IWorld>	 !bottalk
[13:18:31] <wm-bot>	 wm-bot-helping: hi :)
[13:18:31] <wm-bot-helping>	 Dunno!
[13:18:32] <wm-bot-helping>	 Sorry, I don' know, wm-bot.
[13:18:35] <petan|wk>	 I think this channels is not going to be boring again
[13:18:39] <petan|wk>	 channel
[13:19:08] <petan|wk>	 I need to make my bot repond to pings
[13:19:11] <petan|wk>	 then I just loop it
[13:19:16] <petan|wk>	 it's gonna be fun
[13:19:18] <IWorld>	 petan|wk: I can give you owner rights on the bot
[13:19:25] <petan|wk>	 it's ok
[13:19:29] <IWorld>	 (if you need that)
[13:19:29] <petan|wk>	 I am pretty happy being no one
[13:19:41] <IWorld>	 @list BadWords
[13:19:41] <wm-bot-helping>	 IWorld: add, list, and remove
[13:19:53] <petan|wk>	 in fact it makes me less responsible for mess caused by that bot :)
[13:20:15] <petan|wk>	 @dance
[13:20:15] <wm-bot-helping>	 I don't know, petan|wk.
[13:20:19] <petan|wk>	 eh
[13:20:23] <IWorld>	 @dance is fun
[13:20:25] <wm-bot-helping>	 Gotcha.
[13:20:49] <IWorld>	 @dance
[13:20:49] <wm-bot-helping>	 It has been said that dance is fun, IWorld
[13:21:07] <petan|wk>	 @ryan is yay
[13:21:08] <wm-bot-helping>	 Okay!
[13:21:12] <petan|wk>	 @ryan
[13:21:13] <wm-bot-helping>	 It has been said that ryan is yay, petan|wk
[13:21:26] <petan|wk>	 Ryan is going to like it
[13:22:18] <IWorld>	 petan|wk: do you know the wiki link regex?
[13:22:22] <petan|wk>	 ?
[13:22:28] <petan|wk>	 hm...
[13:22:47] <petan|wk>	 how do I make that bot say something what begins with "word"
[13:22:51] <IWorld>	 [[Test]] -> https://meta.wikimedia.org/wiki/Test
[13:23:01] <petan|wk>	 that isn't regex
[13:23:19] <IWorld>	 @re
[13:23:19] <wm-bot-helping>	 IWorld: (re <regexp> <text>) -- If <regexp> is of the form m/regexp/flags, returns the portion of <text> that matches the regexp. If <regexp> is of the form s/regexp/replacement/flags, returns the result of applying such a regexp to <text>
[13:23:41] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[13:23:42] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[13:23:42] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[13:23:42] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[13:23:42] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[13:23:56] <petan|wk>	 link = "https://meta.wikimedia.org/wiki/" + strreplace(strreplace(link, "]]", ""), "[[", "")
[13:24:02] <petan|wk>	 maybe like that
[13:24:22] <IWorld>	 oh
[13:24:39] <IWorld>	 let me see...
[13:24:57] <Hydriz>	 wtf...
[13:25:00] <petan|wk>	 :o
[13:25:13] <petan|wk>	 @hydriz is what the fuck
[13:25:14] <wm-bot-helping>	 Okay, petan|wk.
[13:25:16] <petan|wk>	 @hydriz
[13:25:17] <wm-bot-helping>	 I guess hydriz is what the fuck, petan|wk
[13:25:21] <petan|wk>	 :O
[13:25:33] <Hydriz>	 now, how do we play
[13:25:40] <petan|wk>	 @play
[13:25:40] <wm-bot-helping>	 Dunno!
[13:26:10] <petan|wk>	 !moo is on blah
[13:26:10] <wm-bot>	 Key was added!
[13:26:11] <wm-bot-helping>	 Gotcha, petan|wk.
[13:26:13] <petan|wk>	 !moo
[13:26:13] <wm-bot-helping>	 I guess moo is on blah, petan|wk
[13:26:13] <wm-bot>	 on blah
[13:26:19] <IWorld>	 @forget hydriz
[13:26:20] <wm-bot-helping>	 Gotcha!
[13:26:25] <petan|wk>	 !moo del
[13:26:25] <wm-bot-helping>	 I don't know.
[13:26:25] <wm-bot>	 Successfully removed moo
[13:26:38] <petan|wk>	 I will find out how to loop them
[13:26:43] <petan|wk>	 then it's gonna be fun
[13:26:49] <IWorld>	 wow
[13:26:56] <petan|wk>	 !bottalk
[13:26:56] <wm-bot>	 wm-bot-helping: hi :)
[13:26:57] <wm-bot-helping>	 Sorry, I don' know!
[13:26:58] <wm-bot-helping>	 Wish I knew, wm-bot.
[13:27:10] <Hydriz>	 hmm, there is a Todo plugin...
[13:27:17] <IWorld>	 where?
[13:27:17] <Hydriz>	 @help
[13:27:17] <wm-bot>	 Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.1.4 source code licensed under GPL and located in wikimedia svn
[13:27:18] <wm-bot-helping>	 Hydriz: (help [<plugin>] [<command>]) -- This command gives a useful description of what <command> does. <plugin> is only necessary if the command is in more than one plugin.
[13:27:25] <IWorld>	 @list
[13:27:25] <wm-bot-helping>	 IWorld: Admin, AttackProtector, BadWords, Channel, ChannelLogger, ChannelStats, Config, Infobot, Internet, Limiter, Math, Misc, News, Owner, Plugin, RSS, Scheduler, Seen, ShrinkUrl, Status, String, Time, Topic, URL, User, Utilities, and Web
[13:27:29] <Hydriz>	 ah
[13:27:38] <Hydriz>	 you didn't install it?
[13:27:53] <IWorld>	 @load ToDo
[13:27:53] <wm-bot-helping>	 IWorld: Error: No plugin named "ToDo" exists.
[13:28:29] <Hydriz>	 small caps
[13:28:31] <Hydriz>	 Todo
[13:28:43] <IWorld>	 @load Todo
[13:28:44] <wm-bot-helping>	 IWorld: The operation succeeded.
[13:28:46] <IWorld>	 oh
[13:28:49] <IWorld>	 @list
[13:28:49] <wm-bot-helping>	 IWorld: Admin, AttackProtector, BadWords, Channel, ChannelLogger, ChannelStats, Config, Infobot, Internet, Limiter, Math, Misc, News, Owner, Plugin, RSS, Scheduler, Seen, ShrinkUrl, Status, String, Time, Todo, Topic, URL, User, Utilities, and Web
[13:29:03] <IWorld>	 @list todo
[13:29:03] <wm-bot-helping>	 IWorld: add, change, remove, search, setpriority, and todo
[13:29:12] <IWorld>	 @todo add Test
[13:29:12] <wm-bot-helping>	 IWorld: The operation succeeded.  (Todo #1 added)
[13:30:23] <Hydriz>	 @whoami
[13:30:23] <wm-bot>	 You are trusted identified by name .*@wikimedia/.*
[13:30:23] <wm-bot-helping>	 Hydriz: I don't recognize you.
[13:30:31] <Hydriz>	 :(
[13:30:34] <IWorld>	 @re \\[\\[*(.*?) *(?:\\|.*?)?\\]\\] https://labsconsole.wikimedia.org/wiki/$1
[13:30:35] <wm-bot-helping>	 Error: I tried to send you an empty message.
[13:30:46] <IWorld>	 wm-bot-helping: badbot!
[13:30:46] <wm-bot-helping>	 Dunno.
[13:31:12] <IWorld>	 @whoami
[13:31:12] <wm-bot>	 You are trusted identified by name .*@wikimedia/.*
[13:31:12] <wm-bot-helping>	 IWorld: IWorld
[13:31:36] <IWorld>	 @calc 4+4+4+4+4+4+4+4+4*29398372/228983
[13:31:37] <wm-bot-helping>	 IWorld: 545.546804785
[13:31:49] <Hydriz>	 hmm
[13:32:00] <petan|wk>	 @calc 2/0
[13:32:00] <wm-bot-helping>	 petan|wk: Error: float division by zero
[13:32:34] <petan|wk>	 @calc 55555555555555555555555555555555555555555555555555555*88888888888888885553746564576573
[13:32:35] <wm-bot-helping>	 petan|wk: 4938271604938270494945031241368847675688637111754739147559349703750680158276983717888
[13:32:43] <petan|wk>	 @calc 55555555555555555555555555555555555555555555555555555*888888888888888855537465645765738888888888855537465888888888885553746588888888888555374658888888888855537465
[13:32:43] <wm-bot-helping>	 petan|wk: 49382716049382707584629154422551573733067758848303569588878534451787435990709699536041710531962783048107969007391369007863700634305029530621424254104274848448512
[13:32:47] <petan|wk>	 damn
[13:32:58] <petan|wk>	 it knows long numbers heh
[13:33:09] <Hydriz>	 whoa
[13:33:13] <Hydriz>	 lets try...
[13:33:16] <Hydriz>	 @calc 6/2(2+1)
[13:33:16] <wm-bot-helping>	 Hydriz: Error: Something in there wasn't a valid number.
[13:33:34] <IWorld>	 @calc 1+1(2+1)
[13:33:35] <wm-bot-helping>	 IWorld: Error: Something in there wasn't a valid number.
[13:33:38] <Hydriz>	 what is?
[13:33:52] <IWorld>	 @calc 1+1(2*1)
[13:33:52] <wm-bot-helping>	 IWorld: Error: Something in there wasn't a valid number.
[13:33:56] <Hydriz>	 @calc help
[13:33:56] <wm-bot-helping>	 Hydriz: Error: 'help' is not a defined function.
[13:33:58] <IWorld>	 @calc 1*1(2+1)
[13:33:58] <wm-bot-helping>	 IWorld: Error: Something in there wasn't a valid number.
[13:34:01] <IWorld>	 @help calc
[13:34:01] <wm-bot>	 Unknown command type @commands for a list of all commands I know
[13:34:01] <wm-bot-helping>	 IWorld: (calc <math expression>) -- Returns the value of the evaluated <math expression>. The syntax is Python syntax; the type of arithmetic is floating point. Floating point arithmetic is used in order to prevent a user from being able to crash to the bot with something like '10**10**10**10'. One consequence is that large values such as '10**24' might not be exact.
[13:34:03] <petan|wk>	  @calc 1+1*(2*1)
[13:34:06] <petan|wk>	 hehe
[13:34:08] <petan|wk>	 I got it
[13:34:25] <Hydriz>	 @calc 6/2*(2+1)
[13:34:25] <wm-bot-helping>	 Hydriz: 9
[13:34:29] <IWorld>	 @calc 2^10
[13:34:29] <wm-bot-helping>	 IWorld: Error: Something in there wasn't a valid number.
[13:34:30] <petan|wk>	 @calc 1+1*(2*1)
[13:34:31] <Hydriz>	 lulz
[13:34:31] <wm-bot-helping>	 petan|wk: 3
[13:34:35] <Hydriz>	 1, or 9
[13:34:36] <petan|wk>	 ?
[13:34:43] <petan|wk>	 that bot doesn't know match
[13:34:45] <petan|wk>	 math
[13:34:53] <Hydriz>	 @calc 1+1
[13:34:54] <wm-bot-helping>	 Hydriz: 2
[13:34:59] <petan|wk>	 oh, no it's true heh
[13:35:03] <IWorld>	 @calc 2*2*2*2*2*2*2*2*2*2
[13:35:03] <wm-bot-helping>	 IWorld: 1024
[13:35:17] <Hydriz>	 does it accept x?
[13:35:21] <petan|wk>	 @cal (1+1)*(2/99999999999999999999999999999999999)
[13:35:21] <wm-bot-helping>	 Dunno!
[13:35:25] <petan|wk>	 @cal (1+1)*(2/99999999999999999999999999999999999)
[13:35:25] <wm-bot-helping>	 I don't know.
[13:35:28] <Hydriz>	 @calc 2x+6
[13:35:29] <wm-bot-helping>	 Hydriz: Error: invalid syntax (<string>, line 1)
[13:35:30] <petan|wk>	 @calc (1+1)*(2/99999999999999999999999999999999999)
[13:35:31] <wm-bot-helping>	 petan|wk: 0
[13:35:32] <Hydriz>	 :(
[13:35:36] <petan|wk>	 lies!
[13:35:56] <Hydriz>	 man, today's log file would be huge :P
[13:36:09] <Hydriz>	 @list
[13:36:09] <wm-bot-helping>	 Hydriz: Admin, AttackProtector, BadWords, Channel, ChannelLogger, ChannelStats, Config, Infobot, Internet, Limiter, Math, Misc, News, Owner, Plugin, RSS, Scheduler, Seen, ShrinkUrl, Status, String, Time, Todo, Topic, URL, User, Utilities, and Web
[13:36:21] <Hydriz>	 @time
[13:36:22] <wm-bot-helping>	 Hydriz: 02:36 PM, March 19, 2012
[13:36:26] <Hydriz>	 lies!
[13:36:39] <Hydriz>	 @help Web
[13:36:39] <wm-bot>	 Unknown command type @commands for a list of all commands I know
[13:36:39] <wm-bot-helping>	 Hydriz: Error: There is no command "web".
[13:36:51] <Hydriz>	 hmm, this bot is confusing me
[13:37:28] <IWorld>	 @help web
[13:37:28] <wm-bot>	 Unknown command type @commands for a list of all commands I know
[13:37:28] <wm-bot-helping>	 IWorld: Error: There is no command "web".
[13:37:34] <IWorld>	 @list Web
[13:37:34] <wm-bot-helping>	 IWorld: doctype, fetch, headers, netcraft, size, title, urlquote, and urlunquote
[13:37:42] * Hydriz  facepalms

[13:37:49] <Hydriz>	 @web fetch
[13:37:49] <wm-bot-helping>	 Hydriz: (web fetch <url>) -- Returns the contents of <url>, or as much as is configured in supybot.plugins.Web.fetch.maximum. If that configuration variable is set to 0, this command will be effectively disabled.
[13:37:56] <Hydriz>	 heheheh
[13:38:10] <IWorld>	 nono
[13:38:24] <IWorld>	 I think that writes the source in channel.
[13:38:26] <petan|wk>	 @calc 3.5^33333333333333333333.5
[13:38:26] <wm-bot-helping>	 petan|wk: Error: Something in there wasn't a valid number.
[13:38:30] <petan|wk>	 @calc 3.5^33333333333333333333
[13:38:30] <wm-bot-helping>	 petan|wk: Error: Something in there wasn't a valid number.
[13:38:31] <petan|wk>	 what
[13:38:50] <petan|wk>	 it doesn't know simple math
[13:38:58] <Hydriz>	 @web fetch http://www.google.com
[13:38:58] <wm-bot-helping>	 Hydriz: Error: This command is disabled (supybot.plugins.Web.fetch.maximum is set to 0).
[13:39:03] <IWorld>	 heh
[13:39:06] <Hydriz>	 :(
[13:39:11] <petan|wk>	 I hope it's not going to echo source code of page
[13:39:13] * Hydriz  felt evil for once

[13:39:24] <IWorld>	 @title http://www.google.de/
[13:39:25] <wm-bot-helping>	 IWorld: Google
[13:39:29] <IWorld>	 @title http://www.googledhwuidebiwbfofee.de/
[13:39:31] <wm-bot-helping>	 IWorld: T-Online Navigationshilfe
[13:39:35] <IWorld>	 heh
[13:39:52] <petan|wk>	 @title http://labs.wikimedia.beta.wmflabs.org/wiki/Special:Smack
[13:39:53] <Hydriz>	 lets see if I can do digging
[13:39:55] <wm-bot-helping>	 petan|wk: HTTP Error 404: Not Found
[13:39:58] <petan|wk>	 :D
[13:40:05] <Hydriz>	 @title http://localhost
[13:40:05] <wm-bot-helping>	 Hydriz: Error: 'http://localhost' is not a valid http url.
[13:40:08] <Hydriz>	 :(
[13:40:14] <Hydriz>	 @title 127.0.0.1
[13:40:16] <wm-bot-helping>	 Hydriz: That URL appears to have no HTML title.
[13:40:19] <IWorld>	 nono
[13:40:21] <Hydriz>	 :O
[13:40:30] <IWorld>	 @title http://127.0.0.1/huggle/
[13:40:31] <wm-bot-helping>	 IWorld: Index of /huggle
[13:40:35] <IWorld>	 @title http://127.0.0.1/huggle/hugglewa
[13:40:36] <petan|wk>	 @title http://en.wikipedia.org/wiki/Main_Page?action=delete
[13:40:36] <wm-bot-helping>	 IWorld: HTTP Error 404: Not Found
[13:40:38] <wm-bot-helping>	 petan|wk: Unable to proceed - Wikipedia, the free encyclopedia
[13:40:42] <Hydriz>	 heh
[13:40:43] <petan|wk>	 :o
[13:40:47] <Hydriz>	 @title 208.80.153.211
[13:40:53] <wm-bot-helping>	 Hydriz: Bots - WMF Labs
[13:41:10] <petan|wk>	 @title http://en.wikipedia.org/wiki/Main_Page?action=move
[13:41:11] <Hydriz>	 and move your ass there, wm-bot-helping
[13:41:12] <wm-bot-helping>	 petan|wk: Error
[13:41:18] <petan|wk>	 @title http://en.wikipedia.org/wiki/Main_Page?action=raw
[13:41:19] <wm-bot-helping>	 petan|wk: That URL appears to have no HTML title within the first 4096 bytes.
[13:41:25] <Hydriz>	 lol
[13:41:26] <petan|wk>	 hehe
[13:41:37] <petan|wk>	 @title http://en.wikipedia.org/wiki/Main_Page?action=edit
[13:41:39] <wm-bot-helping>	 petan|wk: View source for Main Page - Wikipedia, the free encyclopedia
[13:41:44] <IWorld>	 @doctype http://www.google.de/
[13:41:45] <wm-bot-helping>	 IWorld: That URL has no specified doctype.
[13:41:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[13:41:51] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[13:41:51] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[13:41:51] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[13:41:52] <Hydriz>	 is there ip lookups?
[13:41:59] <petan|wk>	 @title http://en.wikipedia.org/wiki/User:IWorld?action=rollback
[13:42:01] <wm-bot-helping>	 petan|wk: Unable to proceed - Wikipedia, the free encyclopedia
[13:42:04] <Hydriz>	 hmm
[13:42:07] <IWorld>	 @dns 208.80.153.211
[13:42:14] <wm-bot-helping>	 IWorld: Host not found.
[13:42:17] <Hydriz>	 @list misc
[13:42:18] <wm-bot-helping>	 Hydriz: apropos, help, last, list, more, ping, source, tell, and version
[13:42:20] <IWorld>	 sry
[13:42:30] <IWorld>	 @dns http://www.wikimedia.org/
[13:42:31] <wm-bot-helping>	 IWorld: Host not found.
[13:42:35] <petan|wk>	 @title http://en.wikipedia.org/wiki/User:IWorld?action=purge
[13:42:37] <wm-bot-helping>	 petan|wk: User:IWorld - Wikipedia, the free encyclopedia
[13:42:49] <petan|wk>	 @title http://en.wikipedia.org/wiki/User:IWorld?action=edit
[13:42:51] <wm-bot-helping>	 petan|wk: Editing User:IWorld - Wikipedia, the free encyclopedia
[13:42:53] <petan|wk>	 :o
[13:43:14] <IWorld>	 @unload Web
[13:43:15] <wm-bot-helping>	 IWorld: The operation succeeded.
[13:43:29] <Hydriz>	 :(
[13:43:29] <IWorld>	 @title http://en.wikipedia.org/wiki/User:IWorld?action=purge
[13:43:29] <wm-bot-helping>	 Wish I knew!
[13:43:32] <petan|wk>	 @title http://en.wikipedia.org/wiki/User:IWorld?action=submit
[13:43:32] <wm-bot-helping>	 I don't know.
[13:43:42] <Hydriz>	 wm-bot-helping: don't wish
[13:43:43] <wm-bot-helping>	 Wish I knew!
[13:43:46] <petan|wk>	 @title http://en.wikipedia.org/wiki/User:IWorld?action=submit
[13:43:47] <wm-bot-helping>	 No idea.
[13:43:48] <Hydriz>	 troll zzz
[13:44:00] <IWorld>	 <IWorld> @unload Web ...
[13:44:09] <petan|wk>	 @unload bot
[13:44:09] <wm-bot-helping>	 petan|wk: Error: You don't have the owner capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
[13:44:16] <petan|wk>	 @unload core
[13:44:16] <wm-bot-helping>	 petan|wk: Error: You don't have the owner capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
[13:44:20] <petan|wk>	 damn
[13:44:23] <IWorld>	 @unload bot
[13:44:24] <wm-bot-helping>	 IWorld: Error: There was no plugin bot.
[13:44:29] <IWorld>	 @unload core
[13:44:29] <wm-bot-helping>	 IWorld: Error: There was no plugin core.
[13:44:41] <petan|wk>	 @exec sudo reboot
[13:44:41] <wm-bot-helping>	 Dunno!
[13:44:59] <IWorld>	 No, no! Don't reboot my pc ;)
[13:45:04] <Hydriz>	 LOL
[13:45:13] <Hydriz>	 if it works lol
[13:45:38] <IWorld>	 @config
[13:45:39] <wm-bot-helping>	 IWorld: (config <name> [<value>]) -- If <value> is given, sets the value of <name> to <value>. Otherwise, returns the current value of <name>. You may omit the leading "supybot." in the name if you so choose.
[13:45:44] <IWorld>	 thats the core
[13:45:56] <IWorld>	 and unload config runs...
[13:46:15] <IWorld>	 @bastion
[13:46:15] <wm-bot-helping>	 bastion is bastion is down., IWorld
[13:46:20] <Damianz>	 ...
[13:46:20] <wm-bot-helping>	 Sorry, I don' know, Damianz.
[13:46:27] <Damianz>	 That bot is fucking annoying.
[13:46:32] <Hydriz>	 zzz
[13:46:38] <petan|wk>	 @!bottalk is blah
[13:46:38] <wm-bot-helping>	 10-4!
[13:46:43] <petan|wk>	 @!bottalk is on blah
[13:46:43] <wm-bot-helping>	 ... but !bottalk is blah ...
[13:46:50] <petan|wk>	 @!bottalk
[13:46:50] <wm-bot-helping>	 Somebody said !bottalk is blah, petan|wk
[13:47:00] <petan|wk>	 @!bottalk
[13:47:00] <wm-bot-helping>	 !bottalk is blah, petan|wk
[13:47:00] <wm-bot>	 You are not autorized to perform this, sorry
[13:47:07] <Hydriz>	 lol
[13:47:09] <IWorld>	 :O
[13:47:12] <petan|wk>	 almost there
[13:47:25] <Hydriz>	 @list time
[13:47:26] <wm-bot-helping>	 Hydriz: at, ctime, elapsed, seconds, time, and until
[13:47:27] <petan|wk>	 @!bottalk
[13:47:27] <wm-bot-helping>	 It has been said that !bottalk is blah, petan|wk
[13:47:29] <petan|wk>	 @!bottalk
[13:47:29] <wm-bot-helping>	 It has been said that !bottalk is blah, petan|wk
[13:47:30] <petan|wk>	 @!bottalk
[13:47:30] <wm-bot-helping>	 It has been said that !bottalk is blah, petan|wk
[13:47:32] <petan|wk>	 meh
[13:47:41] <Hydriz>	 @at
[13:47:41] <IWorld>	 @forget !bottalk
[13:47:41] <wm-bot-helping>	 Got it.
[13:47:42] <wm-bot-helping>	 Hydriz: (at <time string>) -- Returns the number of seconds since epoch <time string> is. <time string> can be any number of natural formats; just try something and see if it will work.
[13:48:06] <Hydriz>	 @at 2021039
[13:48:10] <wm-bot-helping>	 Hydriz: An error has occurred and has been logged. Please contact this bot's administrator for more information.
[13:48:12] <petan|wk>	 @whoami
[13:48:12] <wm-bot>	 You are admin identified by name .*@wikimedia/Petrb
[13:48:12] <wm-bot-helping>	 petan|wk: I don't recognize you.
[13:48:22] <Hydriz>	 error...
[13:48:22] <IWorld>	 @register
[13:48:22] <wm-bot-helping>	 IWorld: Error: That operation cannot be done in a channel.
[13:48:43] <IWorld>	 @seconds
[13:48:43] <wm-bot-helping>	 IWorld: (seconds [<years>y] [<weeks>w] [<days>d] [<hours>h] [<minutes>m] [<seconds>s]) -- Returns the number of seconds in the number of <years>, <weeks>, <days>, <hours>, <minutes>, and <seconds> given. An example usage is "seconds 2h 30m", which would return 9000, which is '3600*2 + 30*60'. Useful for scheduling events at a given number of seconds in the future.
[13:48:45] <Hydriz>	 don't be evil to me :(
[13:48:51] <IWorld>	 @seconds 20000y
[13:48:51] <wm-bot-helping>	 IWorld: 630720000000
[13:49:01] <Hydriz>	 @list todo
[13:49:02] <wm-bot-helping>	 Hydriz: add, change, remove, search, setpriority, and todo
[13:49:11] <IWorld>	 @todo
[13:49:13] <wm-bot-helping>	 IWorld: Todo for IWorld: #1: Test
[13:49:33] <Hydriz>	 @add Labs must stay online 24/7 for 630720000000 seconds!
[13:49:33] <wm-bot>	 Permission denied
[13:49:33] <wm-bot-helping>	 Hydriz: Error: The command "add" is available in the BadWords, News, RSS, Scheduler, Todo, and Topic plugins.  Please specify the plugin whose command you wish to call by using its name as a command before "add".
[13:49:43] <IWorld>	 @todo add ...
[13:49:43] <wm-bot-helping>	 IWorld: The operation succeeded.  (Todo #2 added)
[13:49:50] <Hydriz>	 @todo add Labs must stay online 24/7 for 630720000000 seconds!
[13:49:51] <wm-bot-helping>	 Hydriz: Error: You must be registered to use this command. If you are already registered, you must either identify (using the identify command) or add a hostmask matching your current hostmask (using the "hostmask add" command).
[13:49:55] <Hydriz>	 wtf
[13:50:08] <Hydriz>	 @identify
[13:50:09] <wm-bot-helping>	 Hydriz: Error: That operation cannot be done in a channel.
[13:50:17] <IWorld>	 Hydriz: type "register <name> <pwd>" in query
[13:50:25] <petan|wk>	 ok I think it's time
[13:50:30] <petan|wk>	 to move this bot
[13:50:35] <petan|wk>	 to own channel :P
[13:50:43] <Hydriz>	 @identify
[13:50:44] <wm-bot-helping>	 Hydriz: Error: That operation cannot be done in a channel.
[13:50:53] <IWorld>	 Hydriz: query
[13:50:54] <petan|wk>	 because it's probably getting to be annoying to some people
[13:50:58] <Hydriz>	 @todo add Labs must stay online 24/7 for 630720000000 seconds!
[13:50:58] <wm-bot-helping>	 Hydriz: The operation succeeded.  (Todo #1 added)
[13:51:04] <Hydriz>	 @todo
[13:51:05] <wm-bot-helping>	 Hydriz: Todo for Hydriz: #1: Labs must stay online 24/7 for 630720000000...
[13:51:14] <Hydriz>	 :(
[13:51:17] <hashar>	 or those commands should send a notice
[13:51:18] <IWorld>	 petan|wk: you can register one
[13:51:18] <hashar>	 to the user
[13:51:22] <hashar>	 instead of spamming the channel
[13:51:31] <petan|wk>	 ^
[13:51:43] <IWorld>	 .quit
[13:51:47] <hashar>	 changing the channel is fine to :-]
[13:52:05] <IWorld>	 The bot is quitted
[13:52:10] <Hydriz>	 :(
[13:52:13] <petan|wk>	 yay
[13:52:18] <petan|wk>	 !bottalk del
[13:52:19] <wm-bot>	 Successfully removed bottalk
[13:52:29] <Hydriz>	 we will mourn
[13:52:37] <petan|wk>	 and moan
[13:53:29] <IWorld>	 wm-bot-helping: part #wikimedia-labs
[13:53:41] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[13:53:41] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[13:53:41] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[13:53:41] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[13:53:41] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[13:54:08] <IWorld>	 petan|wk: I haven't got access to bots, so that I can't run it 24/7.
[13:54:27] <petan|wk>	 hm... I can't imagine we could stand it 24/7 in this channel :D
[13:54:52] <petan|wk>	 it was here 30 mins and whole channel was spammed like hell
[13:54:57] <Damianz>	 sudo killall -u iworld
[13:55:03] <petan|wk>	 :)
[13:55:17] <petan|wk>	 Damianz: don't do that on solaris though
[13:55:30] <Damianz>	 True
[13:56:17] <IWorld>	 If I type "@unload Infobot" the bot don't spams. OR we don't type these commads like "@bottalk"!!
[13:56:54] <petan|wk>	 but if we do that... what's the bot here for... :D
[13:57:24] <petan|wk>	 I know, sometimes I need to do @calc 2+1
[13:57:38] <petan|wk>	 :P
[13:57:50] <Hydriz>	 isn't Infobot part of wm-bot?
[13:57:56] <petan|wk>	 yes it is
[13:58:01] <Hydriz>	 then we can have best of both worlds :P
[13:58:09] <petan|wk>	 ?
[13:58:15] <IWorld>	 Hydriz: #wmbot-playing
[14:00:10] <IWorld>	 @convert 1 GB to MB
[14:00:10] <wm-bot-helping>	 IWorld: 1024
[14:00:23] <IWorld>	 If you need that :P
[14:00:41] <Hydriz>	 @calc 2^8
[14:00:43] <wm-bot-helping>	 Hydriz: Error: Something in there wasn't a valid number.
[14:02:22] <IWorld>	 the infobot is off
[14:02:25] <IWorld>	 @test
[14:02:25] <wm-bot-helping>	 IWorld: Error: "test" is not a valid command.
[14:05:45] <IWorld>	 petan|wk: oh, the supybot need 16648 kB ram
[14:11:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[14:11:52] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[14:11:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[14:11:52] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[14:23:41] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[14:23:41] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[14:23:42] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[14:23:42] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[14:23:42] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[14:40:05] <labs-home-wm>	 03/19/2012 - 14:40:05 - Updating keys for jgreen
[14:41:51] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[14:41:51] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[14:41:52] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[14:41:52] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[14:53:09] <petan|wk>	 hi sumanah
[14:53:25] <petan|wk>	 re mailing, I was confused by name of mailing list, it wasn't clear it's a private list
[14:53:41] <petan|wk>	 I thought it's for developers :)
[14:53:45] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[14:53:45] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[14:53:45] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[14:53:45] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[14:53:46] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[14:53:51] <petan|wk>	 it say: Development and operation
[14:54:52] <petan|wk>	 althought it was a bit confusing that there were rarely any dev related things
[15:05:15] <gerrit-wm>	 New patchset: Jgreen; "ah ha, gid not groups . . .  yay." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3277
[15:05:26] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3277
[15:11:55] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[15:11:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[15:11:56] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[15:11:56] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[15:19:26] <gerrit-wm>	 New review: Jgreen; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3277
[15:19:29] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3277
[15:21:12] <IWorld>	 petan|wk: are you here?
[15:21:34] <petan|wk>	 yes
[15:22:24] <Damianz>	 petan|wk is everywhere, he IS the internets.
[15:23:07] <IWorld>	 petan|wk: hugglewa-w1 has problems
[15:23:19] <petan|wk>	 !instances
[15:23:19] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Help:Instances
[15:23:21] <wm-bot-helping>	 Wish I knew!
[15:23:30] <Damianz>	 ...
[15:23:31] <wm-bot-helping>	 I have no idea, Damianz.
[15:23:31] <petan|wk>	 IWorld: can you shut your bot
[15:23:34] <IWorld>	 sec.
[15:23:41] * Damianz  kills it with fire

[15:23:45] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[15:23:45] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[15:23:45] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[15:23:45] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[15:23:45] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[15:23:46] <petan|wk>	 hm
[15:23:57] <IWorld>	 I set it to @ only.
[15:24:02] <petan|wk>	 !blah
[15:24:03] <wm-bot-helping>	 No idea.
[15:24:14] <IWorld>	 sec.
[15:25:36] <IWorld>	 !test
[15:25:36] <wm-bot>	 $1
[15:25:37] <wm-bot-helping>	 I don't know.
[15:25:42] <IWorld>	 sec
[15:26:41] <IWorld>	 !hey
[15:26:45] <IWorld>	 okay
[15:26:49] <IWorld>	 @hey
[15:26:50] <wm-bot-helping>	 I don't know!
[15:26:53] <IWorld>	 okay
[15:35:21] <petan|wk>	 !help | IWorld
[15:35:21] <wm-bot>	 IWorld: !documentation for labs !wm-bot for bot
[15:35:25] <petan|wk>	 ah
[15:35:37] <petan|wk>	 !docs | IWorld 
[15:35:38] <wm-bot>	 IWorld : View complete documentation at https://labsconsole.wikimedia.org/wiki/Help:Contents
[15:35:40] <petan|wk>	 that's how
[15:35:44] <petan|wk>	 you fix wa
[15:36:10] <IWorld>	 I can fix?
[15:36:14] <petan|wk>	 yes
[15:36:16] <petan|wk>	 reboot
[15:36:23] <IWorld>	 aah
[15:36:29] <IWorld>	 one click?
[15:36:37] <petan|wk>	 2 click
[15:36:41] <IWorld>	 ah
[15:37:30] <IWorld>	 Do you are WMF staff?
[15:37:35] <petan|wk>	 no
[15:37:44] <IWorld>	 community staff?
[15:37:53] <petan|wk>	 community member
[15:37:55] <petan|wk>	 maybe
[15:37:56] <Damianz>	 community has staff?
[15:37:58] <IWorld>	 ah
[15:37:58] <petan|wk>	 whatever
[15:38:07] * Damianz  kicks petan|wk out the community into the bos

[15:38:07] <IWorld>	 ok :)
[15:38:10] <petan|wk>	 IWorld: what you need
[15:39:37] <IWorld>	 !log hugglewa Rebooting hugglewa-w1
[15:39:38] <labs-morebots>	 Logged the message, Master
[15:41:04] <gerrit-wm>	 New patchset: Jgreen; "adding manages_homedir" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3278
[15:41:15] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3278
[15:41:38] <gerrit-wm>	 New review: Jgreen; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3278
[15:41:41] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3278
[15:41:52] <IWorld>	 the error is still here, petan|wk 
[15:41:55] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[15:41:55] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[15:41:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[15:41:55] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[15:44:18] * IWorld  is away

[15:53:45] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[15:53:45] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[15:53:45] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[15:53:45] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[15:53:45] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[16:11:55] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[16:11:55] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[16:11:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[16:11:55] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[16:16:54] <gerrit-wm>	 New patchset: Jgreen; "manages_homedir does not equal managehome . . ." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3279
[16:17:06] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3279
[16:17:31] <gerrit-wm>	 New review: Jgreen; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3279
[16:17:33] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3279
[16:20:47] <andrewbogott>	 Jeff_Green:  did you make any progress re: puppet dev using local .pp files?  What with labs being flaky it would be useful for me to be able to work on a local machine.
[16:21:05] <Jeff_Green>	 andrewbogott: i didn't get very far on that no
[16:21:13] <andrewbogott>	 ok
[16:21:21] <Jeff_Green>	 it seemed as though I was going to have to rip out a huge chunk of the config to make it work
[16:21:38] <Jeff_Green>	 so I turned back to littering git/gerrit/email with trival changes
[16:23:45] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[16:23:45] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[16:23:46] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[16:23:46] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[16:23:46] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[16:29:10] <IWorld>	 re and back
[16:29:53] <IWorld>	 petan|wk: hugglewa-w1 is still down...
[16:31:14] <IWorld>	 petan|wk: [54/100]: url error [[Errno 101] Network is unreachable]
[16:32:50] <jeremyb>	 heh, trivial changes
[16:34:00] <jeremyb>	 i also want to learn more about labs puppet workflow. e.g. for 1) change in puppet test branch, 2) test on labs, 3) merge to prod, 4) test in prod
[16:34:46] <Damianz>	 that's kinda changing
[16:35:14] <Damianz>	 right now it's merge into test -> test on labs -> can't go into production
[16:35:24] <jeremyb>	 i don't follow
[16:35:27] <Damianz>	 It will be test in local branch -> merge changes into test -> cherry pick into production
[16:35:44] <Damianz>	 The test branch will be 'stable' as far as labs is concerned with each project having a seperate branch.
[16:35:58] <Damianz>	 The test and production branches as they are now as broken and can't be merged.
[16:36:07] <jeremyb>	 how do you explain me pushing to test and having it end up in prod without me doing anything further?
[16:39:30] <andrewbogott>	 someone might have merged or cherry-picked it by hand.
[16:41:35] <IWorld>	 wow: http://nagios.wikimedia.org/
[16:41:55] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[16:41:55] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[16:41:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[16:41:55] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[16:42:20] <gerrit-wm>	 New patchset: Andrew Bogott; "The noca-core ppas are deprecated; switch to openstack-ppa." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3280
[16:42:31] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3280
[16:42:38] <IWorld>	 labs-home-wm: I know that hugglewa-w1 is down
[16:42:40] <IWorld>	 ...
[16:43:34] <gerrit-wm>	 New review: Andrew Bogott; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3280
[16:43:37] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3280
[16:53:45] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[16:53:45] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[16:53:45] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[16:53:46] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[16:53:46] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[16:58:29] <gerrit-wm>	 New patchset: Jgreen; "switching to systemuser" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3281
[16:58:40] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3281
[16:58:51] <gerrit-wm>	 New review: Jgreen; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3281
[16:58:54] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3281
[17:11:30] <gerrit-wm>	 New patchset: Jgreen; "forgot to remove inherits misc::mwlib::groups from misc::mwlib::users class" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3282
[17:11:42] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3282
[17:11:47] <gerrit-wm>	 New review: Jgreen; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3282
[17:11:50] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3282
[17:11:55] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[17:11:55] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[17:11:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[17:11:56] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[17:23:31] <IWorld>	 Can anyone make hugglewa-w1 up?
[17:23:45] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[17:23:45] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[17:23:46] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[17:23:46] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[17:23:46] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[17:24:56] <gerrit-wm>	 New patchset: Jgreen; "undoing inadvertent edit on generic-definitions.pp" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3283
[17:25:07] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3283
[17:25:24] <gerrit-wm>	 New review: Jgreen; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3283
[17:25:26] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3283
[17:41:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[17:41:55] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[17:41:55] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[17:41:55] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[17:53:45] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[17:53:45] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[17:53:46] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[17:53:46] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[17:53:46] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[18:11:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[18:11:55] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[18:11:56] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[18:11:56] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[18:13:18] <marktraceur>	 *All* of labs is down?
[18:14:29] <marktraceur>	 I'm thinking something may have changed, I'm getting an error when I start mongodb
[18:14:37] <marktraceur>	 "cannot open directory /data/db: Transport endpoint is not connected"
[18:15:18] <marktraceur>	 Just food for thought, I'll wait for more stable times before really troubleshooting
[18:17:32] <IWorld>	 yes, that's right
[18:17:42] <IWorld>	 We have some issues.
[18:18:30] <marktraceur>	 Unnnnderstandable
[18:18:41] <marktraceur>	 Not pressing, I do development on my laptop
[18:19:06] <IWorld>	 which instance?
[18:23:45] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[18:23:45] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[18:23:46] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[18:23:46] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[18:23:46] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[18:25:42] <IWorld>	 @ marktraceur 
[18:25:56] <marktraceur>	 Oh, sorry
[18:26:01] <marktraceur>	 orgcharts-dev
[18:26:08] <IWorld>	 let me see
[18:27:06] <IWorld>	 hmm
[18:28:28] <IWorld>	 @ marktraceur: in orgcharts-dev we have only problems with dpkg-check.
[18:28:39] <IWorld>	 dpkg-check
[18:28:40] <IWorld>	 	
[18:28:40] <IWorld>	 	CRITICAL 	2012-03-19 18:26:40 	10d 14h 23m 10s 	4/4 	DPKG CRITICAL dpkg reports broken packages 
[18:28:57] <marktraceur>	 Yeah, that hasn't been a problem up to now
[18:29:13] <marktraceur>	 Every apt-get command returns something similar
[18:29:19] <IWorld>	 ok
[18:29:22] <marktraceur>	 But mongo was working before
[18:29:47] <IWorld>	 In hugglewa-w1 we have more problems.... :-(
[18:36:18] <marktraceur>	 Well, like I said, I have no issue with working locally, focus on the more pressing issue.
[18:36:35] <marktraceur>	 And best of luck :)
[18:41:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[18:41:55] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[18:41:55] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[18:41:55] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[18:53:45] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[18:53:45] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[18:53:46] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[18:53:46] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[18:53:46] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[18:58:40] <IWorld>	 hi Ryan_Lane!
[18:58:46] <IWorld>	 We need you!
[18:58:59] <IWorld>	 --> hugglewa-w1 doesn't runs
[18:59:10] <Ryan_Lane>	 yeah, I was having issues with that instance
[19:00:05] <IWorld>	 is it up?
[19:00:42] <IWorld>	 no... :-(
[19:00:56] <IWorld>	 Ryan_Lane: can you fix that?
[19:01:05] <Ryan_Lane>	 I'm going to look at it
[19:01:20] <Ryan_Lane>	 if I can't bring it up, I'll at least try to save its data
[19:01:35] <IWorld>	 ok
[19:01:50] <IWorld>	 .ryan
[19:01:51] <wm-bot-helping>	 I guess ryan is yay, IWorld
[19:02:01] <IWorld>	 :-)
[19:02:12] * IWorld  is away and the bot, too

[19:03:32] <andrewbogott>	 Ryan_Lane:  What are the prospects for me being able to create an instance today?
[19:03:43] <Ryan_Lane>	 apparently brewster is dead
[19:03:56] <Ryan_Lane>	 mark is looking at it. when that comes up, you should be able to create one
[19:05:16] <andrewbogott>	 ok
[19:05:29] <andrewbogott>	 What runs on brewster?
[19:05:55] <mark>	 ubuntu mirror, apt repository
[19:06:24] <andrewbogott>	 Ah, that'd break things :)
[19:11:12] <Ryan_Lane>	 hm. gonna need to bring virt3 down today
[19:11:19] <Ryan_Lane>	 to try to reseat memory
[19:11:55] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[19:11:55] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[19:11:56] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[19:11:56] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[19:14:53] <Jeff_Green>	 Ryan_Lane: is there a tricky way to recreate instance (i.e. reset to factory settings) without literally recreating it? 
[19:14:59] <Ryan_Lane>	 no
[19:15:31] <Ryan_Lane>	 when we switch APIs you'll be able to snapshot
[19:15:49] <Ryan_Lane>	 so, if you knew you were going to do so, you could snapshot the instance, then rollback to the snapshot
[19:15:56] <Jeff_Green>	 right, that'd work
[19:15:56] <Ryan_Lane>	 but, in general it's kind of wasteful
[19:16:04] <Ryan_Lane>	 it's better to create a new instance usually
[19:16:13] <Ryan_Lane>	 is this for puppetization?
[19:16:18] <Jeff_Green>	 well if you're developing manifests it's exactly what you want to do 
[19:16:20] <Jeff_Green>	 yes
[19:16:24] <Ryan_Lane>	 I have a process for this
[19:16:31] <Jeff_Green>	 oh?
[19:16:32] <Ryan_Lane>	 http://ryandlane.com/blog/2011/11/02/a-process-for-puppetization-of-a-service-using-nova/
[19:16:46] <Ryan_Lane>	 I think I have that documented on wikitech as well
[19:17:07] <Jeff_Green>	 it's basically what i'm already doing
[19:17:40] <Ryan_Lane>	 yep. I just treat instances as really disposable and cheap
[19:17:42] <Ryan_Lane>	 like a git branch
[19:17:43] <Ryan_Lane>	 :)
[19:17:55] <Jeff_Green>	 right, except for the part where creating them doesn't always go well
[19:18:11] <Ryan_Lane>	 true. I don't run into that issue very often
[19:18:19] <Ryan_Lane>	 usually 1 out of 10 tries
[19:18:31] <Ryan_Lane>	 brewster being down just now will make that 10/10
[19:18:45] <Ryan_Lane>	 no apt, no build :)
[19:18:51] <Jeff_Green>	 yeah and it was virt1 or whatever before that
[19:19:13] <Ryan_Lane>	 for the failed builds? or the slowness?
[19:19:15] <Jeff_Green>	 i just submitted an instance deletion bug to bugzilla
[19:19:20] <Ryan_Lane>	 it was virt2 for the slowness
[19:19:24] <Jeff_Green>	 ah yeah
[19:19:25] <Jeff_Green>	 that
[19:19:38] <Ryan_Lane>	 I think that was actually due to the gluster problems we were having
[19:19:42] <Jeff_Green>	 interesting
[19:19:51] <Jeff_Green>	 i heard all hell broke loose over the weekend
[19:20:02] <Ryan_Lane>	 we'll see if it does so again now
[19:20:11] <Ryan_Lane>	 I just did what triggered it over the weekend
[19:20:28] <Jeff_Green>	 could any of that explain an instance delete failing now?
[19:20:34] <Ryan_Lane>	 yes
[19:20:39] <Ryan_Lane>	 when you say delete....
[19:20:41] <Ryan_Lane>	 err
[19:20:43] <Ryan_Lane>	 failing...
[19:20:51] <Ryan_Lane>	 it actually reports back a failure?
[19:20:54] <Jeff_Green>	 yup
[19:21:11] <Jeff_Green>	 approximately: instance deleted but could not remove dns entry
[19:21:27] <Ryan_Lane>	 oh. that usually means it's already been deleted
[19:21:27] <Jeff_Green>	 and the instance appears up and status=running in the manage instances tool
[19:21:30] <Ryan_Lane>	 yes
[19:21:43] <Ryan_Lane>	 it's likely on virt3, and the api is waiting for that node to come back up
[19:21:50] <Jeff_Green>	 i see
[19:21:53] <Ryan_Lane>	 also, sometimes it takes nova a while to delete the instance
[19:22:05] <Ryan_Lane>	 and rather than it showing deleted status, it shows running, which is a bug in nova
[19:22:08] <Jeff_Green>	 yeah this has been several hours, but the virt3 makes sense
[19:22:16] <Ryan_Lane>	 several hours?
[19:22:26] <Ryan_Lane>	 hm. that's not right then
[19:22:29] <Ryan_Lane>	 what's the instance name?
[19:22:40] <Jeff_Green>	 i-000001ae
[19:22:59] <Jeff_Green>	 there's a new bugzilla ticket with the same details
[19:23:12] <Ryan_Lane>	 it's on virt1
[19:23:15] <Ryan_Lane>	 lemme look at the log
[19:23:35] <Ryan_Lane>	 heh
[19:23:38] <Ryan_Lane>	 it's definitely deleted :D
[19:23:53] <Jeff_Green>	 i think I triple deleted it when it misbehaved the first time
[19:23:57] * Ryan_Lane  nods

[19:24:03] <Ryan_Lane>	 api says otherwise
[19:24:18] <Jeff_Green>	 it says 'running' ?
[19:24:20] <Ryan_Lane>	 Permission denied: '/var/lib/nova/instances/instance-000001ae'
[19:24:22] <Ryan_Lane>	 ah ha
[19:24:27] <Jeff_Green>	 uff
[19:24:32] <Jeff_Green>	 PUPPET!
[19:24:33] <Jeff_Green>	 j/k
[19:24:36] <Ryan_Lane>	 :D
[19:24:47] <Ryan_Lane>	 this is likely due to my creation of a new volume over the weekend
[19:27:11] <Ryan_Lane>	 ah. right
[19:27:15] <Ryan_Lane>	 I'm dumb
[19:27:37] <Ryan_Lane>	 I forgot to change the share's ownership
[19:27:48] <Ryan_Lane>	 it could delete the instance's files, but not its directory
[19:28:12] <Ryan_Lane>	 Jeff_Green: ok. it should be gone now
[19:28:29] <Jeff_Green>	 yup
[19:28:51] <Jeff_Green>	 we're currently unable to make new instances because of brewster, right?
[19:29:36] <Ryan_Lane>	 I think so
[19:29:43] <Jeff_Green>	 ok
[19:29:43] <Ryan_Lane>	 mark: is brewster still fixing itself?
[19:29:52] <mark>	 yes
[19:30:00] <mark>	 it should be up though
[19:30:00] * Ryan_Lane  nods

[19:30:03] <mark>	 perhaps a bit slow due to resync
[19:30:03] <Ryan_Lane>	 ok
[19:30:47] <Ryan_Lane>	 mark: so, I need to run two different versions of gluster, how do I do that without screwing things up in apt?
[19:31:39] <mark>	 two different versions under the same distro? (lucid-wikimedia)
[19:31:42] <Ryan_Lane>	 yes
[19:31:45] <mark>	 ugh
[19:31:57] <mark>	 I think only by making them separate packages (names)
[19:32:04] <Ryan_Lane>	 :(
[19:32:12] <mark>	 anything else is messier
[19:32:15] <Ryan_Lane>	 yeah
[19:32:19] <mark>	 make sure they conflict: eachother
[19:32:40] <Ryan_Lane>	 I'm running the beta for the instance storage
[19:32:46] <Ryan_Lane>	 at the recommendation of the gluster people
[19:33:07] <Ryan_Lane>	 apparently the current stable is not recommended for running virtual machine images on
[19:33:13] <mark>	 although I -think- we can make reprepro accept multiple versions too
[19:33:17] <mark>	 currently it removes all previous versions
[19:33:21] * Ryan_Lane  nods

[19:33:31] <Ryan_Lane>	 then I could tell puppet to ensure a specific version
[19:33:35] <mark>	 our previous apt setup just offered all versions and apt picked the newest
[19:33:40] <Ryan_Lane>	 I'd have to be careful on upgrades, though
[19:33:44] <mark>	 no that's messy
[19:33:47] <mark>	 you should do proper apt pinning
[19:33:51] <Ryan_Lane>	 ah
[19:34:03] <mark>	 that's how apt determines what should get installed
[19:34:04] <mark>	 it's quite powerful
[19:34:11] <mark>	 and puppet can install the apt-pin
[19:34:14] <mark>	 apt::pin-package
[19:34:21] * Ryan_Lane  nods

[19:35:37] <mark>	 i'm gonna get some food now
[19:37:09] <IWorld>	 Can we move gerrit-wm to #wikimedia-git or to #mediawiki?
[19:37:49] <IWorld>	 oh, it's moved :)
[19:41:03] <Ryan_Lane>	 IWorld: no
[19:41:15] <Ryan_Lane>	 changes to the test repo should be reported here
[19:43:09] <IWorld>	 Ryan_Lane: why the command @MediaWikiMeet doesn't runs on #wikimedia-dev?
[19:43:27] <Ryan_Lane>	 I don't know what you are talking about?
[19:43:44] <IWorld>	 look at #wikimedia-dev
[19:45:01] <IWorld>	 there is "upcoming meetings @MediaWikiMeet", but if I type that I can't get an answer ;)
[19:45:09] <Ryan_Lane>	 dude.
[19:45:13] <Ryan_Lane>	 I have no clue
[19:45:21] <Ryan_Lane>	 does this have anything to do with labs?
[19:45:29] <IWorld>	 no ;)
[19:45:33] <Ryan_Lane>	 then don't ask in here
[19:45:36] <IWorld>	 (outtopic)
[19:47:51] <IWorld>	 is the instance backed up?
[19:51:55] <Ryan_Lane>	 no
[19:55:11] <IWorld>	 ok
[20:17:12] <Ryan_Lane>	 things are going to be a little slow while virt3's instances boot
[20:17:34] <IWorld>	 Can we use hugglewa-w1?
[20:17:45] <IWorld>	 Or is the instance died?
[20:17:55] <Ryan_Lane>	 when I get a chance to look at it, I will
[20:18:26] <IWorld>	 :)
[20:23:14] <labs-nagios-wm>	 PROBLEM SSH is now: CRITICAL on deployment-wmsearch deployment-wmsearch output: Connection refused
[20:23:39] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on test-oneiric test-oneiric output: WARNING - load average: 7.24, 8.60, 6.43
[20:23:54] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on nova-daas-1 nova-daas-1 output: CHECK_NRPE: Socket timeout after 10 seconds.
[20:23:54] <labs-nagios-wm>	 PROBLEM Total Processes is now: CRITICAL on deployment-wmsearch deployment-wmsearch output: Connection refused by host
[20:24:49] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on deployment-wmsearch deployment-wmsearch output: CHECK_NRPE: Socket timeout after 10 seconds.
[20:24:59] <labs-nagios-wm>	 PROBLEM Total Processes is now: CRITICAL on nova-daas-1 nova-daas-1 output: CHECK_NRPE: Socket timeout after 10 seconds.
[20:25:09] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 0.11, 2.19, 3.77
[20:25:39] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on deployment-sql deployment-sql output: OK - load average: 0.36, 2.11, 4.15
[20:25:39] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on deployment-nfs-memc deployment-nfs-memc output: OK - load average: 0.13, 2.06, 4.05
[20:25:39] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on deployment-web2 deployment-web2 output: OK - load average: 4.37, 3.76, 4.54
[20:25:39] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on nova-daas-1 nova-daas-1 output: CHECK_NRPE: Socket timeout after 10 seconds.
[20:26:19] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on nova-production1 nova-production1 output: WARNING - load average: 12.85, 13.50, 14.04
[20:26:49] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on incubator-bots2 incubator-bots2 output: OK - load average: 0.30, 2.22, 4.32
[20:27:39] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on nova-daas-1 nova-daas-1 output: DISK OK
[20:28:19] <labs-nagios-wm>	 RECOVERY SSH is now: OK on deployment-wmsearch deployment-wmsearch output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[20:28:49] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on test-oneiric test-oneiric output: CHECK_NRPE: Socket timeout after 10 seconds.
[20:30:09] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on bots-3 bots-3 output: OK - load average: 2.57, 2.91, 4.68
[20:30:29] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on nova-daas-1 nova-daas-1 output: All packages OK
[20:30:39] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on deployment-wmsearch deployment-wmsearch output: OK - load average: 1.97, 2.02, 0.94
[20:30:39] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on test-oneiric test-oneiric output: OK: 92% free memory
[20:30:59] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on deployment-wmsearch deployment-wmsearch output: USERS OK - 0 users currently logged in
[20:31:39] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on deployment-wmsearch deployment-wmsearch output: DISK OK
[20:32:09] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on test-oneiric test-oneiric output: PROCS OK: 70 processes
[20:32:34] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on deployment-wmsearch deployment-wmsearch output: OK: 96% free memory
[20:33:34] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on test-oneiric test-oneiric output: WARNING - load average: 0.97, 4.66, 5.69
[20:33:34] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on nova-daas-1 nova-daas-1 output: OK: 83% free memory
[20:33:54] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on deployment-wmsearch deployment-wmsearch output: PROCS OK: 94 processes
[20:34:34] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on deployment-wmsearch deployment-wmsearch output: All packages OK
[20:34:44] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on nova-daas-1 nova-daas-1 output: PROCS OK: 97 processes
[20:35:57] <IWorld>	 Bye!
[20:36:34] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on nova-daas-1 nova-daas-1 output: WARNING - load average: 0.05, 2.63, 5.66
[20:38:44] <labs-nagios-wm>	 PROBLEM host: essex-6 is DOWN address: essex-6 CRITICAL - Host Unreachable (essex-6)
[20:41:35] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on nova-daas-1 nova-daas-1 output: OK - load average: 0.37, 1.13, 4.16
[20:43:55] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on essex-7 essex-7 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[20:44:35] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on essex-7 essex-7 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[20:45:20] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on essex-7 essex-7 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[20:46:05] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on essex-7 essex-7 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[20:46:20] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on nova-production1 nova-production1 output: OK - load average: 0.02, 0.50, 4.53
[20:46:45] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on essex-7 essex-7 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[20:47:25] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[20:48:05] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[20:48:15] <labs-nagios-wm>	 PROBLEM Total Processes is now: CRITICAL on essex-7 essex-7 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[20:48:20] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[20:48:20] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[20:48:25] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[20:48:25] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[20:48:25] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[20:48:25] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[20:48:35] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[20:50:14] <Ryan_Lane>	 petan: for hugglewa, did you change its networking configuration at all?
[20:50:29] <Ryan_Lane>	 because it boots, but can't connect to the network
[20:50:49] <Ryan_Lane>	 it's like it isn't DHCPing
[20:52:06] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on mobile-enwp mobile-enwp output: OK - load average: 0.75, 0.97, 3.89
[20:52:36] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[20:52:36] <labs-nagios-wm>	 PROBLEM host: dumpster01 is DOWN address: dumpster01 CRITICAL - Host Unreachable (dumpster01)
[20:52:46] <labs-nagios-wm>	 PROBLEM host: feeds is DOWN address: feeds CRITICAL - Host Unreachable (feeds)
[21:05:49] <nimish_g>	 Ryan_Lane: is there something that automatically kills processes running after a given amount of time on labs instances?
[21:06:00] <Ryan_Lane>	 no
[21:06:04] <Ryan_Lane>	 there was downtime this weekend
[21:06:20] <nimish_g>	 anything today (within the last few hours)?
[21:06:22] <Ryan_Lane>	 yes
[21:06:35] <Ryan_Lane>	 I had to bring down a compute node to test its memory (it has two bad dimms)
[21:07:26] <Ryan_Lane>	 I'm going to have to bring it down at least once more when we get the memory in.
[21:08:18] <nimish_g>	 any idea when that is?
[21:08:34] <nimish_g>	 (like, today, tomorrow...?)
[21:08:45] <Ryan_Lane>	 no clue
[21:08:55] <Ryan_Lane>	 however long it takes dell :(
[21:09:25] <Ryan_Lane>	 I'd live migrate all the instances off before shutting it down, but that doesn't work terribly well in openstack right now
[21:09:32] <Ryan_Lane>	 also, all of our compute nodes are overloaded. heh
[21:17:56] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[21:18:06] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[21:18:26] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[21:18:26] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[21:18:26] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[21:18:26] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[21:18:36] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[21:18:46] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[21:18:46] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[21:23:06] <labs-nagios-wm>	 PROBLEM host: dumpster01 is DOWN address: dumpster01 CRITICAL - Host Unreachable (dumpster01)
[21:23:06] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[21:23:06] <labs-nagios-wm>	 PROBLEM host: feeds is DOWN address: feeds CRITICAL - Host Unreachable (feeds)
[21:37:11] <gerrit-wm>	 New patchset: Andrew Bogott; "Added ppa key, switched to oneiric rep for openstack-ppa/latest-milestone-test." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3286
[21:37:23] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3286
[21:37:50] <gerrit-wm>	 New review: Andrew Bogott; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3286
[21:37:52] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3286
[21:47:56] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[21:48:06] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[21:48:26] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[21:48:26] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[21:48:26] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[21:48:46] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[21:48:46] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[21:48:56] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[21:49:16] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[21:53:06] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[21:53:06] <labs-nagios-wm>	 PROBLEM host: dumpster01 is DOWN address: dumpster01 CRITICAL - Host Unreachable (dumpster01)
[21:53:06] <labs-nagios-wm>	 PROBLEM host: feeds is DOWN address: feeds CRITICAL - Host Unreachable (feeds)
[21:54:40] <andrewbogott>	 Ryan_Lane:  Can you coach me through adding a new ppa key to puppet/files/ppa?  I think I know what I need but I don't know where to find the actual key file.
[21:56:12] <andrewbogott>	 It looks like the key is in trusted.gpg but it's a binary key, and the stuff in files/ppa is ascii
[22:18:06] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[22:18:06] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[22:18:26] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[22:18:26] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[22:18:26] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[22:18:46] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[22:18:46] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[22:19:16] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[22:19:26] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[22:21:02] <gerrit-wm>	 New patchset: Andrew Bogott; "Added a key file for ppa:openstack-ppa/latest-milestone-test" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3287
[22:21:14] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/3287
[22:21:24] <gerrit-wm>	 New review: Andrew Bogott; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/3287
[22:21:27] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/3287
[22:23:06] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[22:23:06] <labs-nagios-wm>	 PROBLEM host: dumpster01 is DOWN address: dumpster01 CRITICAL - Host Unreachable (dumpster01)
[22:23:06] <labs-nagios-wm>	 PROBLEM host: feeds is DOWN address: feeds CRITICAL - Host Unreachable (feeds)
[22:38:16] <andrewbogott>	 Ryan_Lane:  NM, I think I sorted it out.
[22:39:26] <labs-nagios-wm>	 PROBLEM host: essex-7 is DOWN address: essex-7 check_ping: Invalid hostname/address - essex-7
[22:45:00] <labs-nagios-wm>	 PROBLEM host: essex-8 is DOWN address: essex-8 check_ping: Invalid hostname/address - essex-8
[22:48:08] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[22:48:28] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[22:48:28] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[22:48:28] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[22:48:38] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[22:48:48] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[22:48:48] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[22:49:18] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[22:49:58] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[22:50:08] <labs-nagios-wm>	 RECOVERY host: essex-8 is UP address: essex-8 PING OK - Packet loss = 0%, RTA = 10.02 ms
[22:53:08] <labs-nagios-wm>	 PROBLEM host: dumpster01 is DOWN address: dumpster01 CRITICAL - Host Unreachable (dumpster01)
[22:53:08] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[22:53:08] <labs-nagios-wm>	 PROBLEM host: feeds is DOWN address: feeds CRITICAL - Host Unreachable (feeds)
[22:53:11] <Ryan_Lane>	 andrewbogott: were you able to successfully create an instance?
[22:53:18] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on essex-8 essex-8 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[22:53:23] <andrewbogott>	 Ryan_Lane:  Yep, seems to be working fine.
[22:53:23] <Ryan_Lane>	 I wonder if I should try to bring the rest of those instances up
[22:53:30] <Ryan_Lane>	 I'm worried it'll kill virt3 again
[22:53:37] <andrewbogott>	 Just now killed one and started a new one, no problems.
[22:53:58] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on essex-8 essex-8 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[22:54:14] <Ryan_Lane>	 cool
[22:54:17] <Ryan_Lane>	 good to hear
[22:54:26] <andrewbogott>	 you replaced the bad memory in virt3?
[22:54:38] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on essex-8 essex-8 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[22:54:40] <Ryan_Lane>	 no. we needed to bring it down to try to reseat it
[22:54:47] <Ryan_Lane>	 they moved the slots for another check
[22:54:52] <Ryan_Lane>	 so, they know for sure we have two bad dimms
[22:54:59] <Ryan_Lane>	 when they come in we'll have to bring virt3 down again
[22:55:11] <Ryan_Lane>	 it sucks that live-migration doesn't work properly
[22:55:18] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on essex-8 essex-8 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[22:56:08] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on essex-8 essex-8 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[22:57:28] <labs-nagios-wm>	 PROBLEM Total Processes is now: CRITICAL on essex-8 essex-8 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[22:58:58] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on essex-8 essex-8 output: OK - load average: 1.61, 1.44, 0.82
[22:59:38] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on essex-8 essex-8 output: USERS OK - 1 users currently logged in
[23:00:26] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on essex-8 essex-8 output: DISK OK
[23:01:06] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on essex-8 essex-8 output: OK: 83% free memory
[23:02:36] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on essex-8 essex-8 output: PROCS OK: 103 processes
[23:08:16] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on essex-8 essex-8 output: All packages OK
[23:18:36] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[23:18:46] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[23:18:56] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[23:19:16] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[23:19:16] <labs-nagios-wm>	 PROBLEM host: testing-ioerror is DOWN address: testing-ioerror CRITICAL - Host Unreachable (testing-ioerror)
[23:19:16] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[23:19:16] <labs-nagios-wm>	 PROBLEM host: ganglia-collector is DOWN address: ganglia-collector CRITICAL - Host Unreachable (ganglia-collector)
[23:19:46] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[23:20:46] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[23:23:28] <labs-nagios-wm>	 PROBLEM host: dumpster01 is DOWN address: dumpster01 CRITICAL - Host Unreachable (dumpster01)
[23:23:28] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[23:23:28] <labs-nagios-wm>	 PROBLEM host: feeds is DOWN address: feeds CRITICAL - Host Unreachable (feeds)
[23:45:15] <labs-nagios-wm>	 PROBLEM host: essex-8 is DOWN address: essex-8 CRITICAL - Host Unreachable (essex-8)
[23:48:15] <labs-nagios-wm>	 RECOVERY host: ganglia-collector is UP address: ganglia-collector PING OK - Packet loss = 0%, RTA = 0.59 ms
[23:49:05] <labs-nagios-wm>	 PROBLEM host: canonical-bridge is DOWN address: canonical-bridge CRITICAL - Host Unreachable (canonical-bridge)
[23:49:25] <labs-nagios-wm>	 PROBLEM host: asher1 is DOWN address: asher1 CRITICAL - Host Unreachable (asher1)
[23:49:45] <labs-nagios-wm>	 PROBLEM host: hugglewa-w1 is DOWN address: hugglewa-w1 CRITICAL - Host Unreachable (hugglewa-w1)
[23:49:45] <labs-nagios-wm>	 PROBLEM host: puppet-lucid is DOWN address: puppet-lucid CRITICAL - Host Unreachable (puppet-lucid)
[23:49:55] <labs-nagios-wm>	 PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1)
[23:50:46] <labs-nagios-wm>	 PROBLEM host: pad2 is DOWN address: pad2 CRITICAL - Host Unreachable (pad2)
[23:50:46] <labs-nagios-wm>	 PROBLEM host: webserver-lcarr is DOWN address: webserver-lcarr CRITICAL - Host Unreachable (webserver-lcarr)
[23:53:36] <labs-nagios-wm>	 PROBLEM host: feeds is DOWN address: feeds CRITICAL - Host Unreachable (feeds)
[23:53:46] <labs-nagios-wm>	 PROBLEM host: dumps-4 is DOWN address: dumps-4 CRITICAL - Host Unreachable (dumps-4)
[23:53:46] <labs-nagios-wm>	 PROBLEM host: dumpster01 is DOWN address: dumpster01 CRITICAL - Host Unreachable (dumpster01)
[23:53:56] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on essex-9 essex-9 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[23:54:36] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on essex-9 essex-9 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[23:55:16] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on essex-9 essex-9 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[23:56:06] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on essex-9 essex-9 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[23:56:46] <labs-nagios-wm>	 RECOVERY host: feeds is UP address: feeds PING OK - Packet loss = 0%, RTA = 0.67 ms
[23:57:26] <labs-nagios-wm>	 PROBLEM Total Processes is now: CRITICAL on essex-9 essex-9 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[23:58:16] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on essex-9 essex-9 output: CHECK_NRPE: Error - Could not complete SSL handshake.