[00:00:05] that's the link to the exe, best to just use that as there's 3 things there that you will need, and there's a 4th that could potentially be useful [00:00:30] Command line. Oh balls. [00:00:32] the download page is http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html - where you can download the individual programs, but just use the all in one installer [00:01:03] Thehelpfulone: once I get everything set up though, it'll all work by itself? [00:01:29] your bots? yes they can be configured to work by themselves [00:02:26] Alright. I have to go, I'll be back in an hour. If you're still around then, we can continue. Sorry [00:03:46] Thehelpfulone: yes, that's the problem [00:03:55] no problem [00:03:59] Is problem [00:04:00] (that was to Sven_Manguard ) [00:06:59] How do I get mysql access on the bots-sql servers? [00:07:13] Ask nicely [00:07:20] please? :) [00:09:08] What for, how much space, required username etc? [00:25:51] sorry, one sec... [00:26:34] just for some cimple bots, 100MB, username 'kaldari' [00:26:59] cimple/simple [00:28:29] the first bots I'm transferring over is HotArticlesBot which only stores subscriptions in the DB, so it needs virtually no space, but I might move some of my Commons bots over as well, which need more db storage, thus the 100MB. [01:02:51] Thehelpfulone: I'm back, right when I said I would be. [01:05:43] Okay, wow. Is this *legal* ? [01:06:06] wait, no, encryption isn't illegal in the US I don't think [01:09:21] Thehelpfulone or Damianz or Ryan_Lane (although didn't work end for you all) so do I need to download the individual files from the second link or just click the first link [01:09:49] Sven_Manguard: the first link [01:09:57] I'm off to bed soon but I'll start you of [01:10:00] run the installer [01:10:01] And you trust this to be sage [01:10:07] safe* [01:10:08] yes of course [01:11:02] then run putty gen to generate a private/public key pair [01:11:09] Alright, it's installed [01:11:34] Wait, Thehelpfulone I've heard about Public Keys before (from XKCD) but what are they? [01:12:07] see http://www.unixwiz.net/techtips/putty-openssh.html [01:12:22] that will explain everything [01:13:06] Sven_Manguard: I'll help you tomorrow evening my time (UTC) set this up, but in the mean tie [01:13:07] time [01:13:11] skip out Insure the installation directory is in the command path [01:13:28] do the section that says Create and install a public/private keypair [01:13:45] wa wa wha? [01:14:06] Okay, go to bed and then tomorrow you can help. [01:14:12] Sven_Manguard: can you wait until tomorrow for this? I can do it step by step with you as I've set it up myself recently [01:14:17] mhmm [01:14:23] night night [01:14:25] :) [01:14:38] * Sven_Manguard wanders off in a daze and finds himself at his Xbox [01:14:47] ah, there we go, familiarity [01:18:09] * Damianz is bored [01:22:10] Damianz: you know Java? [01:22:40] I can give you a pile of bots and let you mull over them if you know Java [01:23:42] * Sven_Manguard holds out my hand to engage in thumb war with Damianz [01:23:49] * Sven_Manguard realizes that there's no way that could work [01:23:53] I don't particularly know java apart from hacking on it and bordem != lots of free time [01:24:07] bordem == non entertaining lots of work to do [01:24:45] you're working? at 9:30 PM east coast 6:30 PM west coast? [01:32:28] 2.30 gmt [01:33:00] And yes, I pretty much work in one capacity or another for like 20hours a day :P [02:36:59] paravoid: /srv/org/wikimedia/controller/wikis/1.18wmf1/extensions/OATHAuth [02:40:07] 05/03/2012 - 02:40:07 - Creating a home directory for faidon at /export/home/openstack/faidon [02:41:07] 05/03/2012 - 02:41:07 - Updating keys for faidon [02:49:56] no credentials found for your account. can we add rotating BZ quips to that message? [04:44:57] !log deployment-prep [deployment-dbdump] did a checkpoint `git commit -a` on deploymentprep-conf (/usr/local/apache/common) (locally not pushed anywhere) because there were lots of changes on disk but not in the repo. but didn't add any new files to the repo. (so there's still stuff reported uncommited by `git status`) [04:44:58] Logged the message, Master [04:47:14] !log deployment-prep [deployment-dbdump] changed all refs to IPs of prod hosts nfs-home and ekrem to be deployment-feed instead. and commited that to the local repo. (again not pushed anywhere yet) [04:47:15] Logged the message, Master [04:48:09] !log deployment-prep [deployment-dbdump] (that was to address per complaints about beta simplewiki appearing in #cvn-simplewikis on freenode) [04:48:10] Logged the message, Master [04:59:10] !log deployment-prep [deployment-dbdump] apt-get installed 'ack' [04:59:11] Logged the message, Master [05:02:28] ack > grep ? [05:02:38] read about it a few times, never tried it [05:02:55] btw, i found otu that you did that by reading my enotif ;P [05:02:58] out* [05:03:45] !log deployment-prep [deployment-dbdump] apt-get purge ack; - On ubuntu ack is "ack-grep" which was already installed [05:03:46] Logged the message, Master [05:04:11] what's ack then? [05:04:39] haha, kanji code converter/checker [05:12:39] jeremyb: eh.. so.. how am I supposed to make a change to CommonSettings.php on labs? I've done it in the past on prototype.wikimedia.org but permissions are different here [05:12:45] I can't as myself, and I can't under sudo [05:12:48] both denied access [05:19:07] Krinkle-away: where did you try? [05:19:09] jeremyb: btw, https://labsconsole.wikimedia.org/w/index.php?title=Nova_Resource%3ADeployment-prep%2FSAL&action=historysubmit&diff=3396&oldid=3395 [05:19:39] the docs say don't do anything on the apaches, do everything on -dbdump [05:19:42] jeremyb: deployment-dbdump, /usr/local/apache/common-local/wmf-config/CommonSettings.php [05:19:43] so that's where i did it [05:19:49] yep [05:19:57] oh, i did on common/ not common-local/ [05:20:02] same thing [05:20:03] alias [05:20:13] * jeremyb logs back in [05:20:38] Do you change user or something? [05:20:44] didn't [05:20:56] just straight vim/emacs/nano w/e ? [05:21:02] vim [05:22:06] Whatever editor I use I get [ Error writing CommonSettings.php: Permission denied ] [05:22:22] also tried cp to my ~/ , edit there (works), and cp back [05:22:29] on the cp back, same error: [ Error writing CommonSettings.php: Permission denied ] [05:23:03] I can sudo and change the chmod, but that doesn't make sense [05:23:43] give me more context. your cd/pushd cmds, pwd, vim cmd, etc. [05:24:14] krinkle$ ssh deployment-dbdump.pmtpa.wmflabs [05:24:37] cd /usr/local/apache/common-local/wmf-config/ [05:24:53] nano CommonSettings.php [05:24:55] (edit) [05:25:02] ^X [05:25:06] [ Error writing CommonSettings.php: Permission denied ] [05:25:50] i just wrote. tried nano even [05:27:47] $ id jeremyb [05:27:47] uid=2127(jeremyb) gid=550(svn) groups=1042(project-bots),1054(project-bastion),1070(project-search),1072(project-deployment-prep),1106(project-upload-wizard),1122(project-otrs),1146(project-swift),1156(project-swift3),550(svn) [05:27:54] $ id krinkle [05:27:55] uid=2008(krinkle) gid=550(svn) groups=1038(project-testswarm),1042(project-bots),1054(project-bastion),1066(project-gerrit),1072(project-deployment-prep),1152(project-resourceloader2),550(svn) [05:28:13] no obvious issues [05:28:41] same group (svn0 [05:31:22] Krinkle: stat teh thing that you're trying to write [05:32:05] ? [05:32:30] Krinkle: `stat path/to/file/i/passed/to/nano` [05:32:38] oh [05:32:58] ok [05:33:10] jeremyb: Anything in particular? [05:33:16] Access: (0664/-rw-rw-r--) Uid: ( 1002/mwdeploy) Gid: ( 1075/ depops) [05:33:17] pastebin? or /msg [05:33:25] yeah, that's the same thing i have [05:33:42] i have an idea [05:34:11] I'm all ears :0 [05:34:32] btw, this change has no rush. just an E_NOTICE that has been bugging me for a bit [05:34:45] thanks for the help :) [05:35:08] if/when I get it to work I'll be able to help out more on labs (which I intend to) [05:35:18] bingo [05:35:20] figured it out [05:35:41] extract(bingo) [05:36:51] $ for box in deployment-{nfs-memc,dbdump}.pmtpa.wmflabs; do echo -n "$box: "; ssh $box getent group depops; done 2>/dev/null [05:36:54] deployment-nfs-memc.pmtpa.wmflabs: depops:x:1075:petrb,jeremyb,mah,hashar,johnduhart,catrope [05:36:57] deployment-dbdump.pmtpa.wmflabs: depops:x:1075:petrb,hashar,mah,johnduhart,platonides,werdna [05:37:11] oh [05:37:14] hm [05:37:25] since when is that managed per instance? [05:37:31] I thought it was per group / project on labs [05:37:35] I mean.. I can ssh into it [05:37:45] so, what matters here is your uid on the nfs client and your group on the nfs server [05:38:17] that particular group happens to be local. but the gid might overlap with nfs [05:38:27] ugh. [05:38:32] can you or I fix it? [05:38:42] sure [05:45:01] aight, will check out tomorrow [05:45:10] where are you going? ;) [06:02:08] !log deployment-prep [deployment-prep, deployment-nfs-memc] ran `for u in catrope hashar jeremyb johnduhart krinkle mah petrb platonides werdna; do sudo usermod -a -G depops $u; done`; krinkle was unable to modify files in wmf-config and I thought i saw why he couldn't but couldn't see why '''I''' could. turned out the groups on nfs-memc were the important ones and I was there. synced the 2 boxes with eachother and added krinkle to the list. som [06:02:10] Logged the message, Master [06:03:52] !log deployment-prep [deployment-prep, deployment-nfs-memc] …some other deployment-prep boxes have different depops groups. (one empty with a different gid than the rest. one is same gid but just has petrb) [06:04:19] labs-morebots: ping? [06:04:31] damnit. is it the ellipsis? [06:05:01] !log deployment-prep ping [06:05:02] Logged the message, Master [06:05:15] !log deployment-prep [deployment-prep, deployment-nfs-memc] some other deployment-prep boxes have different depops groups. (one empty with a different gid than the rest. one is same gid but just has petrb) [06:05:15] Logged the message, Master [06:27:00] Change on 12mediawiki a page Wikimedia Labs/Terms of use was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=531986 edit summary: [06:43:35] !log deployment-prep hashar: hashar: bug 36441, added ErrorDocument 404 [06:43:37] Logged the message, Master [06:46:22] !log dumps Mounted goodies on every host. [06:46:23] Logged the message, Master [07:12:29] jeremyb: hey [07:16:48] hey hashar Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument [07:16:57] it doesn't work [07:17:07] Ryan_Lane: is there a way to make sudo doesn't need pw [07:17:25] we badly need it on beta [07:17:49] hey :) [07:18:04] petan|wk: for? [07:18:04] had to commute from home to coworking place [07:18:23] Ryan_Lane: we need to be able to execute commands on all instances from one [07:18:29] I do plan on switching sudo to nopasswd, but I'm also changing ALL to the project group [07:18:30] ssh to all and run sudo [07:18:50] petan|wk: works for me : http://en.wikipedia.beta.wmflabs.org/IDONOTEXIST [07:19:03] I need to make some changes for this to work safely [07:19:04] if we put it to for xxx do ssh blah sudo blah it crashes [07:19:08] petan|wk: that gives me a nice 404 error page and redirects me to /wiki/IDONOTEXIST [07:19:20] if you give nopasswd to ALL, you give system services sudo too [07:19:29] which is the same as running all services as root [07:19:38] Ryan_Lane: why system services are in sudoers [07:19:48] they shouldn't be able to run sudo [07:19:52] on my servers they can't [07:19:59] why they can here [07:20:06] ALL means all users [07:20:24] that's why I'm going to switch it to the project group [07:20:49] aha [07:20:59] Ryan_Lane: for example on deployment not ALL users have access [07:21:03] we specified roles [07:21:27] so, for that, you can set NOPASSWD: ALL in the command line [07:21:28] ALL have access only on dbdump [07:21:33] rather than just ALL [07:21:39] that's what I did result was that sudo didn't work at all [07:21:44] just don't do that on any role that has ALL set for users [07:22:03] whatever I did it told me I am not allowed to run that [07:22:11] probably some syntax [07:22:13] problem [07:22:17] dunno [07:22:17] could be [07:22:21] NOPASSWD: ALL [07:22:25] ^^ that should work [07:22:26] I did NOPASSWD: ALL to command line [07:22:28] that didn't work [07:22:29] hm [07:22:34] I'll need to see how it works [07:23:23] I didn't check how the produced file looked [07:23:32] produced file? [07:23:33] I would need to know local root pw [07:23:39] puppet create /etc/sudoers [07:23:39] what produced file? [07:23:42] no [07:23:46] not anymore [07:23:48] I didn't know how did it look [07:23:50] it's in ldap [07:23:59] right but there must be some /etc/sudoers [07:24:00] no [07:24:05] sudo pulls from ldap [07:24:07] that's config file of sudo [07:24:22] ok but how did you tell to sudo it should pull it from ldap [07:24:30] where is the NOPASSWD part [07:24:36] if not in sudoers file [07:24:40] in ldap? [07:24:40] it's all in ldap [07:24:43] eh [07:24:45] right [07:24:54] actually I tried to change it by hand in sudoers and it worked [07:25:12] yeah. it could be that you need to specify it differently in ldap [07:25:17] hm [07:25:20] I'll figure it out and let you know [07:25:29] are we fine using the name of « betalabs » ? :-D [07:25:34] yes [07:26:04] will amend my changes and send them to review so :-] [07:28:27] should I rename it then [07:28:44] meh [07:28:56] I don't see why the links need to change [07:29:02] err [07:29:02] urls [07:29:28] petan|wk: you know it isn't possible to have a project not renamed at minimum 3 times, right? [07:29:40] welcome to the inside joke of projects at WMF [07:29:44] project not [07:29:53] I mean the dns [07:30:00] we didn't rename project so far [07:30:03] I'd prefer to not mess with the dns [07:30:12] we already changed deployment to beta [07:30:20] yeah. let's stick with beta [07:30:22] ok [07:30:28] so hashar what did you want to change then [07:30:39] betalabs is new name of what [07:30:40] it has wmflabs in the domain name. what's the point in adding labs in there again? [07:30:48] I don't know [07:30:48] The two hardest problems in computer science: 1. naming things; 2. cache invalidation; 3. off-by-one errors. [07:30:55] hashar wanted to change it ask him [07:31:09] when we refer to it, i think we should refer to it as beta in labs [07:31:14] I wanted to have a short and useful name to refers to the beta project [07:31:21] jerith: I usualy name variables a b x etc [07:31:23] and got it as betalabs :-] [07:31:24] :D [07:31:25] it's currently beta ;) [07:31:42] so now I can give a useful name to my puppet classes [07:31:58] why do the puppet classes need to be specific to beta? [07:32:01] they should be generic [07:32:02] hashar: ok so I changed the category in bugzilla [07:32:19] a *lot* of the puppet work is already done, it just needs to be applied [07:32:23] and maybe tweaked a little [07:32:35] New patchset: Hashar; "classes for deployment preparation project (beta)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5790 [07:32:50] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5790 [07:33:12] New review: Hashar; "patchset 3 uses « betalabs » instead of « beta »" [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/5790 [07:33:26] I'm really confused by this [07:33:36] wmf deployment-prep (betalabs) [07:33:40] hashar: hm? [07:33:47] we shouldn't be using labs project specific puppet configuration [07:33:58] Ryan_Lane: what you mean [07:34:09] Ryan_Lane: we need to be able to approve changes to puppet for projects [07:34:10] this is a clone of production [07:34:18] or puppet is not going to be useable [07:34:20] use the classes from production [07:34:26] those classes are just about installing a few specifics packages (joe, ack-grep) and an attempt to document the 'devops' group [07:34:28] make changes to them when they aren't generic enough [07:34:52] group which is not that useful btw since its not our primary group :-D [07:35:01] hashar: is that name ok [07:35:05] hashar: wmf deployment-prep (betalabs) [07:35:09] why not use project-deployment-prep, rather than devops/ [07:35:10] that's what is now in bugzilla [07:35:26] petan|wk: whatever name :-] that one sounds good yes [07:35:31] if you are in the project, you're assumed to be able to deploy, right? [07:35:36] I don't know Ryan, devops was there already :( [07:35:37] I don't know who inserted wmf there, but I like using both [07:35:42] deployment-prep betalabs [07:35:55] so that it's clear which project in console it is [07:36:00] either way, don't make any puppet config specific to the project [07:36:08] puppet config is supposed to be generic [07:36:23] !realm | hashar [07:36:23] hashar: $realm is a variable used in puppet to determine which cluster a system is in. See also $site. [07:36:25] :o [07:36:26] if you need to do something labs specific, use $realm [07:36:31] the problem is that we will probably end up having to write a lot of production classes which will take time to validate [07:36:52] whereas adding specific labs classes let us quickly document our changes and implements the labs faster [07:36:53] !site [07:36:53] $site is a variable used in puppet to determine which datacenter a system is in. See also $realm. [07:36:56] it's bad enough that beta isn't using the production puppet classes [07:36:58] yay [07:37:02] datacenter? [07:37:10] if we start writing all new classes, they'll *never* be similar [07:37:25] Ryan_Lane: we need to make a new variable to $PROJECTNAME [07:37:37] so that I can get a project name in shell [07:37:41] hashar: what I'm saying is, nearly everything you need is already done [07:37:49] hashar: it just needs to be tweaked [07:38:02] see how it's done in production, and apply that to beta [07:38:11] petan|wk: yeah [07:38:18] I saw the bug :) [07:38:21] that is what I did for the syslog server. I tweaked the production class so we can choose the remote syslog IP [07:38:48] but I know for sure that ack-grep joe or tree will not make it in production since I already tried to get them installed on fenari :-] [07:38:59] and I still find them useful, hence the new class [07:39:03] wtf is joe [07:39:09] petan|wk: an editor [07:39:23] New review: Ryan Lane; "Please don't make classes specific to a labs project." [operations/puppet] (test); V: 0 C: -2; - https://gerrit.wikimedia.org/r/5790 [07:39:25] is it better than vim? [07:39:29] as for the "devops" groups, we could most probably use the default wmflabs one [07:39:44] what I am missing though is the ability to change the users primary group [07:39:54] we are all in svn(550) right now :-] [07:40:03] still, it shouldn't be specific to the project [07:40:20] make them a class, and import them where needed [07:40:26] but generic classes [07:40:48] stuff such as: package::ack-grep package::joe etc... [07:40:59] then I "just": include package::ack-grep [07:41:00] ? [07:41:43] yes [07:42:25] I think we do it in generic.pp for the most part right now? [07:42:38] indeed [07:43:07] !log nagios root: aptitude upgrade [07:43:08] Logged the message, Master [07:44:55] New review: Hashar; "Following a discussion with Ryan, we should use the default group project-deployment-prep instead of..." [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/5790 [07:45:04] Change abandoned: Hashar; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5790 [07:45:19] Ryan_Lane: can we upgrade dist on instances? [07:45:28] is it supported? [07:45:31] ubuntu 12 [07:45:41] I think you use it on prod [07:45:49] umm [07:45:51] I'd wait on that [07:45:56] I haven't tested if upgrades work [07:45:58] ok [07:46:03] and definitely don't do it yet on beta [07:46:08] unless you want everything to break [07:46:22] we haven't started upgrading most things in production [07:46:26] right [07:48:53] \o/ nagious [07:48:57] *nagios [07:57:27] Ryan_Lane: before you get to sleep, we will probably have to merge the production branch back in test :-D [07:57:28] probably not something you want to do right now ;-] [07:57:28] hahahahaha [07:57:28] good luck on that [07:57:28] it's so insanely diverged right now [07:59:51] New patchset: Hashar; "class to install the Joe editor" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6466 [08:00:05] New patchset: Hashar; "class to install the ack-grep utility" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6467 [08:00:20] New patchset: Hashar; "class to install the 'tree' utility" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6468 [08:00:30] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6466 [08:00:30] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6467 [08:00:30] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/6468 [08:06:28] New patchset: Hashar; "packages specifics to labs bastions" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6469 [08:06:42] New patchset: Hashar; "class to install the 'tree' utility" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6468 [08:06:56] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6469 [08:06:56] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6468 [08:13:26] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:13:35] New patchset: Hashar; "git ignore /private/" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6471 [08:13:49] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6471 [08:14:46] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [08:16:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [08:27:46] PROBLEM Free ram is now: WARNING on mobile-feeds i-000000c1 output: Warning: 17% free memory [08:37:30] New review: Dzahn; "Joe's Own Editor. http://joe-editor.sourceforge.net/" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/6466 [08:37:33] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6466 [08:42:19] New review: Dzahn; "https://launchpad.net/ubuntu/+source/tree" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/6468 [08:43:26] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [08:43:33] New review: Dzahn; "just saying. did you also know "ytree"?" [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/6468 [08:44:46] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [08:46:25] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 0 C: -1; - https://gerrit.wikimedia.org/r/6467 [08:46:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [08:54:26] New review: Dzahn; "yeah, i guess if $::realm == "labs" { is preferred, as used in lvs.pp. Just on a note, i also see va..." [operations/puppet] (test); V: 0 C: 1; - https://gerrit.wikimedia.org/r/6469 [09:07:46] PROBLEM Free ram is now: CRITICAL on mobile-feeds i-000000c1 output: Critical: 5% free memory [09:12:23] mutante: there is something wrong with puppet checks [09:12:48] I give up merging production in test :-( [09:12:55] toooooo many conflicts to solve [09:13:05] hm [09:13:26] PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2) [09:14:46] PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202) [09:15:27] petan|wk: ok, checking, i see [09:15:57] I think it's problem in nagios [09:16:12] what is the service which listen to passive checks [09:16:16] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/petrb 3353 MB (19% inode=81%): [09:16:17] I think it's down [09:16:36] PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1) [09:22:46] RECOVERY Puppet freshness is now: OK on labs-build1 i-0000006b output: puppet ran at Thu May 3 09:22:43 UTC 2012 [09:23:06] RECOVERY Puppet freshness is now: OK on deployment-backup i-000000f8 output: puppet ran at Thu May 3 09:22:58 UTC 2012 [09:23:16] RECOVERY Puppet freshness is now: OK on dev-solr i-00000152 output: puppet ran at Thu May 3 09:23:09 UTC 2012 [09:23:16] !log nagios starting snmptrapd [09:23:17] Logged the message, Master [09:23:46] RECOVERY Puppet freshness is now: OK on bots-dev i-00000190 output: puppet ran at Thu May 3 09:23:36 UTC 2012 [09:23:48] petan|wk: that is that part i mentioned back then that may not be reboot safe. [09:24:16] RECOVERY Puppet freshness is now: OK on dumps-4 i-0000017d output: puppet ran at Thu May 3 09:24:02 UTC 2012 [09:24:16] RECOVERY Puppet freshness is now: OK on p-b i-000000ae output: puppet ran at Thu May 3 09:24:02 UTC 2012 [09:24:16] RECOVERY Puppet freshness is now: OK on hugglewa-db i-00000188 output: puppet ran at Thu May 3 09:24:07 UTC 2012 [09:24:36] RECOVERY Puppet freshness is now: OK on reportcard2 i-000001ea output: puppet ran at Thu May 3 09:24:25 UTC 2012 [09:25:16] RECOVERY Puppet freshness is now: OK on wikisource-web i-000000fe output: puppet ran at Thu May 3 09:25:14 UTC 2012 [09:25:36] RECOVERY Puppet freshness is now: OK on incubator-bots2 i-00000119 output: puppet ran at Thu May 3 09:25:19 UTC 2012 [09:25:36] RECOVERY Puppet freshness is now: OK on pediapress-ocg1 i-00000233 output: puppet ran at Thu May 3 09:25:22 UTC 2012 [09:25:36] RECOVERY Puppet freshness is now: OK on labs-relay i-00000103 output: puppet ran at Thu May 3 09:25:26 UTC 2012 [09:25:46] RECOVERY Puppet freshness is now: OK on bots-nfs i-000000b1 output: puppet ran at Thu May 3 09:25:38 UTC 2012 [09:25:46] RECOVERY Puppet freshness is now: OK on bz-dev i-000001db output: puppet ran at Thu May 3 09:25:43 UTC 2012 [09:26:06] RECOVERY Puppet freshness is now: OK on building i-0000014d output: puppet ran at Thu May 3 09:25:53 UTC 2012 [09:26:06] RECOVERY Puppet freshness is now: OK on simplewikt i-00000149 output: puppet ran at Thu May 3 09:25:57 UTC 2012 [09:26:16] RECOVERY Puppet freshness is now: OK on mobile-feeds i-000000c1 output: puppet ran at Thu May 3 09:26:14 UTC 2012 [09:26:36] RECOVERY Puppet freshness is now: OK on firstinstance i-0000013e output: puppet ran at Thu May 3 09:26:17 UTC 2012 [09:26:36]