[00:04:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.998047/1.95, alarm hl:np_load_avg=1.146484/2.0, alarm hl:mem_free=279.000000M/350M, alarm hl:available=1/0 [00:05:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:08:37] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:09:47] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 335609 MB (6% inode=36%): [00:13:48] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:14:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9830.000000 [00:15:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:17:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:19:48] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.169922/1.10, alarm hl:np_load_long=1.257812/1.55, alarm hl:mem_free=15388.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.169922/1.00, alarm hl:np_load_long=1.257812/1.50, alarm hl:mem_free=15388.000000M/600M, alarm hl:available=1/0 [00:19:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.931152/1.95, alarm hl:np_load_avg=1.034668/2.0, alarm hl:mem_free=249.000000M/350M, alarm hl:available=1/0 [00:22:49] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:33:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:38:37] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:53:37] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:54:07] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38935 MB (9% inode=99%): [01:00:07] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 67232 MB (16% inode=99%): [01:03:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.302734/1.95, alarm hl:np_load_avg=1.509277/2.0, alarm hl:mem_free=143.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.302734/2.3, alarm hl:np_load_long=1.312988/2.5, alarm hl:cpu=76.300000/98, alarm hl:mem_free=143.000000M/150M, alarm hl:available=1/0 [01:05:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:08:47] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:09:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 335473 MB (6% inode=36%): [01:11:03] محمد الجداوي * [Toolserver-l] Renaming my account [01:12:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.779297/1.95, alarm hl:np_load_avg=1.384766/2.0, alarm hl:mem_free=327.000000M/350M, alarm hl:available=1/0 [01:13:07] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42810 MB (10% inode=99%): [01:14:48] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8608.000000 [01:16:07] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 78361 MB (19% inode=99%): [01:16:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:17:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:32:49] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.219726/1.10, alarm hl:np_load_long=0.655273/1.55, alarm hl:mem_free=15752.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.219726/1.00, alarm hl:np_load_long=0.655273/1.50, alarm hl:mem_free=15752.000000M/600M, alarm hl:available=1/0 [01:33:37] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:33:49] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [01:48:17] / on wolfsbane is WARNING: DISK WARNING - free space: / 6282 MB (20% inode=93%): [02:04:23] Krinkle: You know there's a page on mediawiki.org? [02:04:37] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User:Ninia675g10]]": spam) [02:04:48] Joan: Yes, I link to it from tswiki:mwbot, why ? [02:04:50] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Ninia675g10]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [02:05:09] Krinkle: Oh, just thought it was a bit strange, but I hadn't clicked. :-) [02:05:18] Just a stub for now [02:05:21] It's just listing the group members, I see. You're big into indices lately. [02:05:31] I replied to you on wikitech-l, BTW. [02:05:42] in the weekend I'll fix up some stuff and put mwbot in ts-svn [02:05:50] hi Krinkle [02:05:50] I'm not sure amidaniel should be listed as a maintainer. [02:05:52] and maybe document a bit on how it should run, cause it is broken, again. [02:06:01] he has access [02:06:05] on ts [02:06:07] He kind of abandoned it. I'm not sure he has an active acccount. [02:06:09] account [02:06:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:06:11] w/e [02:06:24] Well, just saying that if someone tries to e-mail him, I've no idea if it'll work. :P [02:06:27] I'm behind on mailinglists [02:06:33] Aren't we all. [02:06:36] he [02:06:37] So much noise, every day. [02:07:23] Krinkle: You should fix the tswiki's spam problem. :D [02:07:39] I've blocked like a dozen spammers in the past few weeks. [02:07:41] maybe, or ts should set up some basic anti-spam stuff [02:07:53] I think it has the Match captcha. [02:07:56] Math, too. [02:08:19] it is a very naked install on a unstable version of mediawiki, stabbed randomly out of trunk [02:08:37] running not on sqlite and not on mysql, so kind of unstable there too (not because of the backend but because almost nobody uses it / less stability naturally) [02:08:37] Heh. [02:08:43] anyway, it works.. [02:08:47] I think it was kind of intended as a way of testing trunk. [02:08:50] Back when Aryeh was around. [02:08:52] And River. [02:09:08] I think it's using Postgres. [02:09:13] yeah, that is nice. But unlike translatewiki.net it isn't really being monitored for testing afaik [02:09:15] You know how contrarian River is. [02:09:19] it is indeed on pg [02:09:40] nor is it often updated :P [02:09:44] No, Aryeh and River and everyone have mostly abandoned it. [02:09:46] It is running on last years svn [02:09:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 335448 MB (6% inode=36%): [02:09:51] Yes. [02:10:29] > [02:10:30] Hello, I request changing my account's name, on toolserver wiki, from "M.gedawy" to "محمد الجداوي". As i have problems with logging in using my current name. Thanks. [02:10:34] > [02:10:37] I'm currently in the long-term process of moving stuff into source control and documenting them, and adding small or big features and rewrites here and there - of my tools and gadgets [02:10:38] what [02:10:47] Aren't you moving to Labs? [02:11:07] well I'm not "moving" to labs, but I am using Wikimedia Labs [02:11:22] I thought Wikimedia was killing the TS. [02:11:28] I think we're all moving to Labs. [02:11:29] but right now, although it is going to be a super set of toolserver (much larger afaik) - right now most of tool servers characteristics aren't there yet [02:11:35] Right. [02:11:39] Yes, that does seem like the long-term plan [02:11:40] Well, mostly DB replication, I think. [02:11:54] I'm not sure what else is missing. [02:12:05] indeed, and some complexity issues [02:13:03] right now the only way to get a project going is on a very low level (openstack project, a group of vm instances, puppetizing it, and the only way to get stuff in the web browser is to either use a local proxy through ssh, or request a public IP and subdomain) [02:13:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.035645/1.95, alarm hl:np_load_avg=1.821289/2.0, alarm hl:mem_free=255.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.035645/2.3, alarm hl:np_load_long=1.459961/2.5, alarm hl:cpu=98.700000/98, alarm hl:mem_free=255.000000M/150M, alarm hl:available=1/0 [02:13:26] but then again, the root-level of the wmf labs project does not align with toolserver [02:13:37] the "tool labs" will probably be a single project in wmf labs [02:13:53] That'd be awkward. [02:13:54] e.g. like wikitools.wmflabs.org/~krinkle/foo [02:14:02] how so? [02:14:35] Have a bunch of small projects and then one huge one? [02:14:38] It seems imbalanced. [02:14:47] No, it depends [02:14:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6240.000000 [02:15:00] It won't go all in there, it depends on what you're working on [02:15:04] It seems like something that you might live with if it came naturally, but I wouldn't plan it that way. [02:15:09] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:15:27] e.g. "mwbot" for instance, will probably not be an account or project at all. Instead we'd put it in the "bots.wmflabs.org" project, and people can get access to it [02:15:48] indeed not planned that way [02:15:53] and won't happen either [02:16:07] but there will likely be a "simple" project where you can just do stuff like on toolserver [02:16:10] entry level for beginners [02:16:14] It kind of depends what the primary migration path for TS users is. [02:16:28] Heh. [02:16:31] simple.wmflabs.org? [02:16:33] without having to create a virtual machines or puppetizing stuff or requesting a public subdomain [02:16:39] * Joan makes the sign of the cross. [02:16:40] Joan: maybe :P [02:17:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:17:10] meh, the main tone I get from labs ops right now is that the last thing they want to do is start any kind of migration [02:17:25] Well, yeah. [02:17:37] The one distinguishing factor is DB replication. [02:17:40] In my opinion. [02:17:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:17:55] So until that's up and stable for a while, it doesn't make sense to think about migration. [02:18:09] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.540039/1.95, alarm hl:np_load_avg=1.712402/2.0, alarm hl:mem_free=246.000000M/350M, alarm hl:available=1/0 [02:18:19] wmflabs is structured very differently. Anyone who sees fit for their stuff is welcome, and once everything has a home, we could call out a "clean up" on toolserver, but I don't think that will be any time soon [02:18:22] there are a lot of things on toolserver that don't need db replication [02:18:52] Sure, but most it just needs basic hosting. [02:19:06] What distinguishes the Toolserver from a VPS is DB replication. [02:19:13] Or shared hosting or whatever. [02:19:20] most of * [02:19:24] one aspect is that wmf labs doesn't have any individual-public space (e.g. no ~krinkle), everything is a project and people can join that project. [02:19:43] although not yet, what I also hope will be encouraged a lot is to put everything in source control [02:19:44] Re: Anyone who sees fit <-- I guess it must be Wikimedia-related? [02:19:54] Or you'll have people setting up StarCraft servers or whatever. [02:19:58] hehe [02:20:00] Whatever people play. [02:20:03] I meant more the other way around [02:20:05] Call of Duty? [02:20:14] if you see a spot in labs where your project fits in, not the other way around. [02:20:21] Ah. [02:20:34] e.g. bots.wmflabs.org is going to get pretty awesome if it is up to me. I have some plans to put up a web based control panel there [02:20:44] You know CVN, right ? [02:20:54] Countervandalism Network? [02:21:07] yes [02:21:11] I'm familiar. Bunch of revert nerds. [02:21:21] haha [02:21:31] well, I happen to be one of the senior staff+bot devs there [02:21:51] Then you know already. :-) [02:22:12] and you maybe confused with the en.wiki wikiproject named similarly (CVU- counter vandalism unit) [02:22:33] No shortage of people willing to revert things. [02:22:43] Or vandalism, for that matter. [02:23:08] What would you be able to control about the bots? [02:23:09] Stalk lists? [02:23:12] Rebooting them? [02:23:18] Renaming them? [02:23:45] yeah, I was shooting towards making a point but something is being awfully slow - brb [02:24:40] k, got it [02:24:49] so here is a shot of what I hacked up for cvn: http://cl.ly/0Y27303e0S1T2g1w0r3W [02:24:56] ugly, I know. but it is about the functionality [02:25:10] for non-devs on the team to log in and stop/start and see debug stuff [02:25:42] so what I'd like to have in the wmflabs-bot project is something like that, but with 1 or 2 more levels of hierarchy in it [02:26:05] What debug stuff? [02:26:14] I run a few IRC bots. You can kill them by PMing !restart. [02:26:27] yeah, and if they timed out, how do you start them again? [02:26:35] I use phoenix to keep them up. [02:26:38] Right [02:26:40] And they ping every minute. [02:26:49] So if they can't reach the network, they kill themselves. [02:26:49] Consired using SGE? [02:26:55] Considered* [02:26:56] Not really. [02:27:03] I don't really trust it. [02:27:16] cron + phoenis works well enough. [02:27:18] phoenix [02:27:35] River always had these big ideas for things like SGE. Then you'd switch and she'd get distracted. [02:27:39] I use cron + consub (SGE), same thing basically [02:27:42] Maintenance is a cruel bitch. [02:27:57] Right, but I'm always worried SGE is going to break and nobody is going to fix it. [02:28:10] Even cron broke at some point when it was changed to Solaris. [02:28:20] crontab + cronsub that is [02:28:21] whatever [02:28:53] anyway, in this control panel developers can add commands, give them a descriptive name. and put them in a certain group. and users with an account can log in and execute those commands in their groups [02:28:54] I'm a big fan of stability. ;-) [02:29:03] not all commands are should-be-always-running-bots [02:29:25] Should be a fun project. :-) [02:30:17] I know at least a few other toolserver projects that have a similar thing [02:30:32] so I think in labs stuff will hopefully be less wheel-reinventing [02:30:53] Maybe. Certain pieces of the TS could certainly use improvement. Like user auth. [02:31:02] Or wiki integration. [02:31:13] I always thought it'd be cool to transclude content from the Toolserver. [02:31:23] yeah, one the the ideas of tool labs is to actually use MediaWiki [02:32:03] so in general (iirc) "we" (early meeting last year about labs) devided toolserver uses in three categories [02:32:31] * bots, just needing hosting and long-running queues like phoenis/sge [02:33:12] * fairly large workflow applications, not tied directly into meidawiki or db-replication. users log in there and do stuff for/with wmf wikis [02:33:34] * QueryPage-like stuff. they query the replicated db based on input and give it back [02:33:47] the last one is for example: https://toolserver.org/~krinkle/OrphanTalk2/ [02:33:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:34:26] and for that 3rd category it'd be nice if, on the wmf-toollabs one could just include a php thing that would offer MediaWiki classes and somehow connections to replicated DB stuff [02:35:06] where I could then use the Database class and SpecialPage to write the stuff and do stuff like this with only a few lines of code. no need to worry about front-end or backend [02:35:45] that will do 3 important things 1) easier for devs, 2) familiar and common front-end for users + i18n possibly, 3) small step to becoming an extension [02:35:50] am I ranting? [02:37:38] No. [02:37:43] k :) [02:37:59] You'd still have to write the query, wouldn't it? [02:38:01] And optimize it. [02:38:14] A missing index is a missing index, at the end of the day. Doesn't matter how you access the data. [02:38:19] yeah [02:38:39] yeah, but that doesn't stop toolserver users to write and use such query [02:38:54] I always thought being able to include content from the Toolserver via a template trasnclusion-like interface would be cool. [02:39:10] Run it through the sanitizer and maybe cache it. [02:39:12] another idea that came along is to maybe have some like stable.toolserver.org again, which would be a wiki hosted on labs with selected tool-extensions [02:39:12] transclusion [02:39:21] where Special:SpecialPages would be the front page [02:40:19] I've done something like that once, I had my logic on the toolserver (a db query), and made the toolserver tool output JSON from PHP, and with a corresponding gadget on the wiki request that and display it as a SpecialPage [02:41:10] e.g Special:Foobar or Special:BlankPage?gadget=foobar and have the gadget look for that, clean out the page, show spinner, make the cross-domain request to the tool with JSON-P, make html, hide spinner and sum pit [02:41:12] dump it* [02:41:37] Right. [02:41:44] I meant more for reader-facing content. [02:41:51] For dynamic content, pretty much. [02:41:55] Right now everything is static. [02:41:58] Which often sucks. [02:41:59] right [02:42:10] Take for example : https://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/specials/SpecialWantedtemplates.php?revision=78786&view=markup [02:42:12] The Toolserver is lagged {{toolserver lag}}. [02:42:21] Instead of a bot updating a page every hour forever. [02:42:21] that is pretty awesome to have as source code, basically just the query [02:42:37] Yes. [02:42:38] everything else is standardized (front-end, i18,
, database, ..) [02:42:43] Right. [02:42:55] MediaWiki as a framework. [02:42:59] Yep [02:43:09] but the install itself wouldn't have a database [02:43:46] probably with some kind of ToolserverExtension with a php factory class to get database objects to certain wiki databses [02:44:00] e.g. WmfWikiDb::get( 'enwiki' ) [02:44:15] and that would be all that's neccecary [02:44:38] and of course built-in ability to drop in a wiki-select drop down menu [02:45:39] I've done something like it for my own tools. I use KrinkleBaseTool for all tools and ToolserverIntuition in many for i18n. [02:46:06] source code of OrphanTalk2 is still more than what it could be (I'd like to have the be auto-generated as well), but here it is https://svn.toolserver.org/svnroot/krinkle/trunk/OrphanTalk2/index.php [02:46:33] kfConnectToolserverDB() stuff like that [02:46:53] River worked on something somewhat similar. [02:47:00] ~reports/ was kind of like that. [02:48:28] / on wolfsbane is WARNING: DISK WARNING - free space: / 6152 MB (20% inode=93%): [02:49:05] kfConnectRRServerByDBName() [02:49:51] but yeah, stuff like that [02:56:14] Cool. :-) [03:00:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3537.000000 [03:06:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:07:58] Load avg. on willow is WARNING: WARNING - load average: 15.48, 15.80, 14.00 [03:08:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.971680/1.95, alarm hl:np_load_avg=1.981934/2.0, alarm hl:mem_free=197.000000M/350M, alarm hl:available=1/0 [03:09:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 333579 MB (6% inode=36%): [03:09:58] Load avg. on willow is OK: OK - load average: 12.69, 14.67, 13.80 [03:12:58] Load avg. on willow is WARNING: WARNING - load average: 16.13, 15.79, 14.42 [03:17:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:17:20] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:17:58] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1798.000000 [03:18:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:33:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:40:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.101074/1.95, alarm hl:np_load_avg=1.455566/2.0, alarm hl:mem_free=220.000000M/350M, alarm hl:available=1/0 [03:47:28] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:48:28] / on wolfsbane is WARNING: DISK WARNING - free space: / 6009 MB (20% inode=93%): [03:53:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.885742/1.95, alarm hl:np_load_avg=1.223145/2.0, alarm hl:mem_free=283.000000M/350M, alarm hl:available=1/0 [04:02:58] Load avg. on willow is WARNING: WARNING - load average: 15.34, 13.83, 12.62 [04:04:58] Load avg. on willow is OK: OK - load average: 13.89, 14.25, 12.96 [04:06:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:07:58] Load avg. on willow is WARNING: WARNING - load average: 16.90, 16.91, 14.34 [04:09:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 333545 MB (6% inode=36%): [04:17:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:17:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.625976/1.95, alarm hl:np_load_avg=1.896484/2.0, alarm hl:mem_free=193.000000M/350M, alarm hl:available=1/0 [04:18:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:20:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:33:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:39:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.039062/1.95, alarm hl:np_load_avg=1.408203/2.0, alarm hl:mem_free=295.000000M/350M, alarm hl:available=1/0 [04:48:30] / on wolfsbane is WARNING: DISK WARNING - free space: / 5885 MB (19% inode=93%): [04:55:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:06:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:10:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 333509 MB (6% inode=36%): [05:10:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.093262/1.95, alarm hl:np_load_avg=1.438476/2.0, alarm hl:mem_free=272.000000M/350M, alarm hl:available=1/0 [05:17:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:18:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:31:28] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:34:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.269531/1.95, alarm hl:np_load_avg=1.500488/2.0, alarm hl:mem_free=194.000000M/350M, alarm hl:available=1/0 [05:34:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:38:38] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:44:29] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:49:29] / on wolfsbane is WARNING: DISK WARNING - free space: / 5711 MB (19% inode=93%): [05:53:09] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:55:19] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:29] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:30] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:39] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:39] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:40] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:40] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:40] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:40] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:40] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:49] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:49] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:49] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:00] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:00] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:00] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:00] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:00] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:00] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:00] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:01] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:09] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:09] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:10] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:10] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:10] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:18] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:19] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:19] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:29] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:39] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [05:56:39] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [05:56:39] Load avg. on z-dat-s3-a is OK: OK - load average: 0.55, 1.70, 2.42 [05:56:39] Load avg. on z-dat-s6-a is OK: OK - load average: 0.55, 1.69, 2.41 [05:56:39] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 96140 MB (23% inode=99%): [05:56:40] / on z-dat-s7-a is OK: DISK OK - free space: / 8353 MB (27% inode=85%): [05:56:40] / on z-dat-s4-a is OK: DISK OK - free space: / 8353 MB (27% inode=85%): [05:56:41] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 87954 MB (21% inode=99%): [05:56:41] Load avg. on z-dat-s7-a is OK: OK - load average: 0.55, 1.69, 2.41 [05:56:42] / on z-dat-s6-a is OK: DISK OK - free space: / 8353 MB (27% inode=85%): [05:56:42] / on z-dat-s3-a is OK: DISK OK - free space: / 8353 MB (27% inode=85%): [05:56:43] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 2190 MB (99% inode=99%): [05:56:43] / on hyacinth is OK: DISK OK - free space: / 8353 MB (27% inode=85%): [05:56:44] /tmp on hyacinth is OK: DISK OK - free space: /tmp 2190 MB (99% inode=99%): [05:56:59] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 167212 MB (17% inode=99%): [05:56:59] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:56:59] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2023 MB (99% inode=99%): [05:57:08] SMTP on z-dat-s7-a is OK: SMTP OK - 0.003 sec. response time [05:57:09] SMF on z-dat-s4-a is OK: OK - all services online [05:57:10] SMF on z-dat-s6-a is OK: OK - all services online [05:57:10] SMF on z-dat-s3-a is OK: OK - all services online [05:57:10] SMF on hyacinth is OK: OK - all services online [05:57:10] SMF on z-dat-s7-a is OK: OK - all services online [05:57:17] SMTP on hyacinth is OK: SMTP OK - 0.009 sec. response time [05:57:29] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:07:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:10:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 333470 MB (6% inode=36%): [06:17:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:19:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:35:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:37:59] Load avg. on willow is WARNING: WARNING - load average: 12.21, 15.52, 14.20 [06:38:59] Load avg. on willow is OK: OK - load average: 10.50, 14.39, 13.88 [06:49:30] / on wolfsbane is WARNING: DISK WARNING - free space: / 5528 MB (18% inode=93%): [06:56:00] Load avg. on willow is WARNING: WARNING - load average: 18.38, 17.05, 14.76 [07:05:19] [[Special:Log/newusers]] create 10 * Heike87dh * (New user account) [07:07:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:11:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 333397 MB (6% inode=36%): [07:13:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.291992/1.95, alarm hl:np_load_avg=1.946777/2.0, alarm hl:mem_free=188.000000M/350M, alarm hl:available=1/0 [07:17:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:19:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:27:39] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:27:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1822.000000 [07:32:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.331055/1.95, alarm hl:np_load_avg=1.340332/2.0, alarm hl:mem_free=227.000000M/350M, alarm hl:available=1/0 [07:36:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:36:39] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:37:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43225 MB (10% inode=99%): [07:42:40] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 68679 MB (16% inode=99%): [07:45:59] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1798.000000 [07:49:40] / on wolfsbane is WARNING: DISK WARNING - free space: / 5289 MB (17% inode=93%): [07:51:00] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1820.000000 [07:52:59] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1792.000000 [08:05:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.719726/1.95, alarm hl:np_load_avg=1.800781/2.0, alarm hl:mem_free=173.000000M/350M, alarm hl:available=1/0 [08:07:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:11:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 331555 MB (6% inode=35%): [08:13:41] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:17:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:18:41] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:19:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:32:41] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:35:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.204590/1.95, alarm hl:np_load_avg=1.319336/2.0, alarm hl:mem_free=324.000000M/350M, alarm hl:available=1/0 [08:36:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:36:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:48:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 318320 MB (5% inode=34%): [08:49:41] / on wolfsbane is WARNING: DISK WARNING - free space: / 5034 MB (16% inode=93%): [09:07:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:17:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:19:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:31:00] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1885.000000 [09:36:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:48:11] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 280972 MB (5% inode=32%): [09:49:41] / on wolfsbane is WARNING: DISK WARNING - free space: / 4749 MB (15% inode=93%): [09:55:39] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:57:19] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:07:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:09:11] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:29] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:29] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:40] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:40] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:49] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:50] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:50] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:59] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:59] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:59] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:59] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:10:19] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [10:10:29] SMTP on z-dat-s7-a is OK: SMTP OK - 7.838 sec. response time [10:10:29] SMTP on z-dat-s3-a is OK: SMTP OK - 9.259 sec. response time [10:10:29] SMF on hyacinth is OK: OK - all services online [10:10:29] SMF on z-dat-s3-a is OK: OK - all services online [10:10:29] SMF on z-dat-s6-a is OK: OK - all services online [10:10:30] SMF on z-dat-s7-a is OK: OK - all services online [10:10:30] SMF on z-dat-s4-a is OK: OK - all services online [10:10:40] SMTP on hyacinth is OK: SMTP OK - 0.002 sec. response time [10:10:40] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:10:59] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:11:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.296875/1.95, alarm hl:np_load_avg=1.826172/2.0, alarm hl:mem_free=160.000000M/350M, alarm hl:available=1/0 [10:14:50] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:17:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.372559/1.95, alarm hl:np_load_avg=1.644531/2.0, alarm hl:mem_free=228.000000M/350M, alarm hl:available=1/0 [10:17:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:19:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:20:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:31:10] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3054.000000 [10:37:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:40:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.740723/1.95, alarm hl:np_load_avg=1.541504/2.0, alarm hl:mem_free=197.000000M/350M, alarm hl:available=1/0 [10:41:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 328071 MB (6% inode=35%): [10:42:09] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317676 MB (5% inode=34%): [10:42:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:49:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4464 MB (14% inode=93%): [11:02:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.795899/1.10, alarm hl:np_load_long=0.722656/1.55, alarm hl:mem_free=16215.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.795899/1.00, alarm hl:np_load_long=0.722656/1.50, alarm hl:mem_free=16215.000000M/600M, alarm hl:available=1/0 [11:06:19] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:08:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:09:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.638672/1.10, alarm hl:np_load_long=0.963867/1.55, alarm hl:mem_free=16338.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.638672/1.00, alarm hl:np_load_long=0.963867/1.50, alarm hl:mem_free=16338.000000M/600M, alarm hl:available=1/0 [11:17:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:19:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:31:09] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3073.000000 [11:37:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:39:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.846680/1.95, alarm hl:np_load_avg=1.039551/2.0, alarm hl:mem_free=342.000000M/350M, alarm hl:available=1/0 [11:42:09] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317545 MB (5% inode=34%): [11:43:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:49:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4103 MB (13% inode=93%): [11:54:51] / on wolfsbane is OK: DISK OK - free space: / 6548 MB (21% inode=93%): [12:06:49] [[Special:Log/newusers]] create 10 * Tsmegan69 * (New user account) [12:08:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:12:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.720703/1.95, alarm hl:np_load_avg=1.280274/2.0, alarm hl:mem_free=323.000000M/350M, alarm hl:available=1/0 [12:13:59] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:17:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:20:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:23:56] 3(updated) [TS-1368] Restore a script file from backup <10https://jira.toolserver.org/browse/TS-1368> [12:31:10] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3600.000000 [12:32:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3656.000000 [12:37:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:43:09] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317456 MB (5% inode=34%): [12:53:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.901367/1.95, alarm hl:np_load_avg=1.120605/2.0, alarm hl:mem_free=291.000000M/350M, alarm hl:available=1/0 [12:55:59] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:00:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.596191/1.95, alarm hl:np_load_avg=1.512695/2.0, alarm hl:mem_free=252.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.596191/2.3, alarm hl:np_load_long=1.285156/2.5, alarm hl:cpu=60.600000/98, alarm hl:mem_free=252.000000M/150M, alarm hl:available=1/0 [13:09:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:16:50] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.736816/1.10, alarm hl:np_load_long=0.800781/1.55, alarm hl:mem_free=917.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.736816/1.00, alarm hl:np_load_long=0.800781/1.50, alarm hl:mem_free=917.000000M/600M, alarm hl:available=1/0 [13:17:18] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.320312/1.10, alarm hl:np_load_long=0.970703/1.55, alarm hl:mem_free=16564.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.320312/1.00, alarm hl:np_load_long=0.970703/1.50, alarm hl:mem_free=16564.000000M/600M, alarm hl:available=1/0 [13:17:50] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [13:18:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:21:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:21:32] [[Special:Log/newusers]] create 10 * Lenny09742 * (New user account) [13:22:19] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:25:19] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.248047/1.10, alarm hl:np_load_long=1.060547/1.55, alarm hl:mem_free=16584.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.248047/1.00, alarm hl:np_load_long=1.060547/1.50, alarm hl:mem_free=16584.000000M/600M, alarm hl:available=1/0 [13:32:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5272.000000 [13:37:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:43:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.741699/1.95, alarm hl:np_load_avg=1.534180/2.0, alarm hl:mem_free=303.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.741699/2.3, alarm hl:np_load_long=1.288086/2.5, alarm hl:cpu=92.400000/98, alarm hl:mem_free=303.000000M/150M, alarm hl:available=1/0 [13:43:11] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317382 MB (5% inode=34%): [13:45:12] Load avg. on willow is WARNING: WARNING - load average: 16.78, 13.52, 11.07 [13:53:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:53:12] Load avg. on willow is OK: OK - load average: 8.73, 14.67, 13.40 [14:09:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:18:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:21:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:24:34] [[User:Lenny09742]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7170&rcid=9526 * Lenny09742 * (+397) (Created page with "Kalkulieren Deinen BMI in weniger wie 30 Sekunden. Bist Du abgesperrt wulstig, zu schlank oder tiefschürfend bis über beide Ohren? Finde es grade unentgeltlich hervor. Das Rec...") [14:32:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5584.000000 [14:37:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:39:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.002441/1.95, alarm hl:np_load_avg=1.045410/2.0, alarm hl:mem_free=236.000000M/350M, alarm hl:available=1/0 [14:43:09] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:43:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317146 MB (5% inode=34%): [14:52:57] 3(created) [MAGNUS-318] Article tree view isn't working or just times out; Magnus' tools; Bug <10https://jira.toolserver.org/browse/MAGNUS-318> (Sarah Stierch) [15:09:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:18:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:22:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:32:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5722.000000 [15:33:10] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1976014s failure: longrun-sol@willow in error state: QERROR as result of job 1976014s failure [15:37:06] [[Special:Log/newusers]] create 10 * Hotelfocus * (New user account) [15:37:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:43:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317066 MB (5% inode=34%): [15:45:30] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.168945/1.10, alarm hl:np_load_long=0.665039/1.55, alarm hl:mem_free=16288.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.168945/1.00, alarm hl:np_load_long=0.665039/1.50, alarm hl:mem_free=16288.000000M/600M, alarm hl:available=1/0 [15:46:30] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:46:38] [[User:Hotelfocus]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7171&rcid=9528 * Hotelfocus * (+1029) (Created page with "Hotel Focus owo dziewiczy kanon w dziedzinie miejskich hoteli biznesowych. Nasza sieć oferuje noclegi w dogodnych punktach takich w zastępstwie kiedy Szczecin, Bydgoszcz, Łó...") [16:09:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:14:54] [[Special:Log/newusers]] create 10 * Gakmo * (New user account) [16:18:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:22:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:33:11] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1976014s failure: longrun-sol@willow in error state: QERROR as result of job 1976014s failure [16:33:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6382.000000 [16:38:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:43:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316941 MB (5% inode=34%): [16:56:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:03:39] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:09:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:18:09] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:18:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:19:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.047363/1.95, alarm hl:np_load_avg=1.134766/2.0, alarm hl:mem_free=311.000000M/350M, alarm hl:available=1/0 [17:20:19] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:22:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:26:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.045410/1.95, alarm hl:np_load_avg=1.063476/2.0, alarm hl:mem_free=330.000000M/350M, alarm hl:available=1/0 [17:33:39] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:34:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6244.000000 [17:38:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:44:20] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316724 MB (5% inode=34%): [17:44:29] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.501953/1.10, alarm hl:np_load_long=0.860352/1.55, alarm hl:mem_free=15954.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.501953/1.00, alarm hl:np_load_long=0.860352/1.50, alarm hl:mem_free=15954.000000M/600M, alarm hl:available=1/0 [17:45:30] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:10:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:13:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.489258/1.95, alarm hl:np_load_avg=1.350586/2.0, alarm hl:mem_free=260.000000M/350M, alarm hl:available=1/0 [18:14:19] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:18:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:22:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:34:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6421.000000 [18:38:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:44:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316624 MB (5% inode=34%): [19:10:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:19:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:23:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:28:39] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:35:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6361.000000 [19:39:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:44:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316413 MB (5% inode=34%): [19:58:19] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:07:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.294434/1.95, alarm hl:np_load_avg=1.708008/2.0, alarm hl:mem_free=363.000000M/350M, alarm hl:available=1/0 [20:10:19] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:10:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:16:58] 3(commented) [OSM-16] Found on ptolemy in dmesg <10https://jira.toolserver.org/browse/OSM-16> (Kai Krueger) [20:19:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:23:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:36:30] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6311.000000 [20:39:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:45:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316262 MB (5% inode=34%): [20:54:21] [[Special:Log/newusers]] create 10 * Salman666 * (New user account) [21:02:29] Load avg. on willow is WARNING: WARNING - load average: 15.77, 15.33, 12.85 [21:03:30] Load avg. on willow is OK: OK - load average: 12.78, 14.43, 12.68 [21:10:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:13:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.496582/1.95, alarm hl:np_load_avg=1.560547/2.0, alarm hl:mem_free=225.000000M/350M, alarm hl:available=1/0 [21:14:20] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:17:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.238281/1.95, alarm hl:np_load_avg=1.462891/2.0, alarm hl:mem_free=236.000000M/350M, alarm hl:available=1/0 [21:19:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:23:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:36:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6403.000000 [21:37:30] Load avg. on willow is WARNING: WARNING - load average: 15.30, 13.99, 12.70 [21:38:30] Load avg. on willow is OK: OK - load average: 12.58, 13.36, 12.55 [21:40:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:42:19] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.327149/1.95, alarm hl:np_load_avg=1.577149/2.0, alarm hl:mem_free=309.000000M/350M, alarm hl:available=1/0: longrun-sol@willow in error state: QERROR as result of job 1977371s failure [21:43:17] [[Special:Log/newusers]] create 10 * Qzwjdf675 * (New user account) [21:45:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316071 MB (5% inode=34%): [22:10:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:12:30] Load avg. on willow is WARNING: WARNING - load average: 20.68, 14.40, 13.23 [22:14:30] Load avg. on willow is OK: OK - load average: 11.37, 13.00, 12.85 [22:19:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:23:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:36:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7593.000000 [22:40:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:42:40] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.009766/1.95, alarm hl:np_load_avg=1.548340/2.0, alarm hl:mem_free=357.000000M/350M, alarm hl:available=1/0: longrun-sol@willow in error state: QERROR as result of job 1977371s failure [22:44:52] [[User:Qzwjdf675]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7172&rcid=9532 * Qzwjdf675 * (+156) (Created page with "Good day this can be a good wonderful web-site you should view. My Site: [http://www.killstress.de/abnehmen-mit-hypnose.html abnehmen Hypnose Erfahrungen]") [22:45:40] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315939 MB (5% inode=34%): [23:09:57] So much spam. [23:10:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:19:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:23:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:36:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8254.000000 [23:40:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:42:49] Sun Grid Engine execd on willow is CRITICAL: longrun-sol@willow in error state: QERROR as result of job 1977371s failure [23:43:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:45:40] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315870 MB (5% inode=34%): [23:50:44] [[Special:Log/block]] block 10 * Merlissimo * (blocked [[02User:Qzwjdf67510]] with an expiry time of infinite (account creation disabled, e-mail blocked): Spamming links to external sites) [23:51:39] [[Special:Log/delete]] delete 10 * Merlissimo * (deleted "[[02User:Qzwjdf67510]]": Spam) [23:56:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.028809/1.95, alarm hl:np_load_avg=1.228027/2.0, alarm hl:mem_free=322.000000M/350M, alarm hl:available=1/0 [23:57:50] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK