[00:01:26] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 76333 MB (18% inode=99%): [00:04:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:05:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 155660 [00:07:24] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:08:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:12:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:16:54] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:32:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:45:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 427342 MB (8% inode=41%): [00:58:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:58:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:05:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:06:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 157397 [01:08:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:13:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:32:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:45:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 434986 MB (8% inode=41%): [01:49:35] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40304 MB (9% inode=99%): [01:55:54] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199027 MB (20% inode=99%): [01:56:35] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22615 MB (5% inode=99%): [01:58:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:58:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:04:35] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 27255 MB (6% inode=99%): [02:05:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:06:34] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 159391 [02:09:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:13:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:46:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 427928 MB (8% inode=41%): [02:58:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:59:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:05:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:06:35] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 159911 [03:09:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:12:27] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:12:39] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:14:27] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [03:19:38] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:33:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:44:27] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [03:45:08] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 258.000000 [03:46:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 427903 MB (8% inode=41%): [03:47:40] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43422 MB (10% inode=99%): [03:59:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:00:39] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 67237 MB (16% inode=99%): [04:00:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:06:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:06:40] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 159937 [04:09:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:12:27] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:33:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:37:09] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:46:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 436554 MB (8% inode=41%): [05:00:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:00:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:06:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:06:41] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 160656 [05:10:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:13:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:24:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43311 MB (10% inode=99%): [05:33:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:36:38] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 67985 MB (16% inode=99%): [05:46:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 427868 MB (8% inode=41%): [06:00:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:00:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:06:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:07:27] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:27] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:27] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:39] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 161859 [06:07:39] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:07:48] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:18] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [06:08:18] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:08:18] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:08:18] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:08:38] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:10:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:13:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:14:28] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:28] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:18] SMTP on z-dat-s6-a is OK: SMTP OK - 0.005 sec. response time [06:33:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:47:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 427804 MB (8% inode=41%): [07:01:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:01:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:06:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:08:00] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 163312 [07:10:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:14:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:34:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:47:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 427578 MB (8% inode=41%): [07:52:28] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:01:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:01:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:06:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:08:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 163905 [08:11:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:14:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:26:58] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:34:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:47:08] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 431555 MB (8% inode=41%): [08:50:58] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:50:58] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:03] [[Special:Log/newusers]] create 10 * Maalaimalar * (New user account) [08:51:10] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:18] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:18] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:18] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:18] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:29] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:38] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:38] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:38] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:38] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:38] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:38] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:38] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:39] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:39] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:40] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:48] / on z-dat-s6-a is OK: DISK OK - free space: / 8219 MB (27% inode=85%): [08:51:48] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 1587 MB (99% inode=99%): [08:51:58] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:52:28] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [08:52:28] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [08:52:30] [[User talk:Maalaimalar]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7388&rcid=9972 * Maalaimalar * (+1226) (Maalaimalar update the Tamil cinema news on Regularly. Day to Day the kollywood latest news has been updated in Maalaimalar.com. ) [08:52:49] Load avg. on z-dat-s6-a is OK: OK - load average: 0.26, 1.42, 2.28 [08:52:49] / on z-dat-s7-a is OK: DISK OK - free space: / 8219 MB (27% inode=85%): [08:52:49] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:52:49] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 87.000000 [08:52:58] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [08:52:58] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:52:58] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [08:52:58] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1574 MB (99% inode=99%): [08:53:10] MySQL slave on z-dat-s3-a is OK: Uptime: 4192339 Threads: 27 Questions: 4429582343 Slow queries: 293088 Opens: 44585965 Flush tables: 1 Open tables: 16385 Queries per second avg: 1056.589 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 257 [08:53:10] SMF on z-dat-s7-a is OK: OK - all services online [08:53:10] Load avg. on z-dat-s4-a is OK: OK - load average: 0.54, 1.41, 2.26 [08:53:10] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 76893 MB (19% inode=99%): [08:53:10] MySQL on z-dat-s6-a is OK: Uptime: 849441 Threads: 15 Questions: 161235750 Slow queries: 73765 Opens: 1433923 Flush tables: 1 Open tables: 2852 Queries per second avg: 189.813 [08:53:10] Load avg. on z-dat-s7-a is OK: OK - load average: 0.59, 1.41, 2.25 [08:53:10] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:53:28] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:53:28] SMF on z-dat-s6-a is OK: OK - all services online [08:54:29] SMTP on z-dat-s7-a is OK: SMTP OK - 0.003 sec. response time [08:54:29] SMTP on z-dat-s6-a is OK: SMTP OK - 0.005 sec. response time [08:54:40] [[Special:Log/newusers]] create 10 * Lexy99 * (New user account) [08:55:57] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:56:38] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:57:09] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:57:18] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:57:28] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:57:28] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:59:02] Dr. Trigon * [Toolserver-l] Anoter SGE question [09:02:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:02:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:05:24] [[User talk:Maalaimalar]] ! 10https://wiki.toolserver.org/w/index.php?diff=7389&oldid=7388&rcid=9974 * Maalaimalar * (+1073) (/* Lalu is a 'tired' man : Sushil Modi */ new section) [09:06:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:08:56] [[User talk:Maalaimalar]] ! 10https://wiki.toolserver.org/w/index.php?diff=7390&oldid=7389&rcid=9975 * Maalaimalar * (+2284) (/* Hazare not aware of the whole matter on PM, says his Team */ new section) [09:08:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 165586 [09:12:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:14:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:30:08] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:08] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:20] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:38] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:38] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:48] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:48] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:49] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:49] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:49] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:49] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:49] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:49] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:49] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:50] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:50] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:51] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:08] Load avg. on z-dat-s3-a is OK: OK - load average: 0.91, 1.21, 1.46 [09:31:19] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 76922 MB (19% inode=99%): [09:31:20] SMF on z-dat-s3-a is OK: OK - all services online [09:31:20] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 116069 MB (11% inode=98%): [09:31:20] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1517 MB (99% inode=99%): [09:31:20] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 1518 MB (99% inode=99%): [09:31:20] SMF on z-dat-s6-a is OK: OK - all services online [09:31:20] Load avg. on z-dat-s4-a is OK: OK - load average: 1.01, 1.22, 1.46 [09:31:21] Load avg. on z-dat-s7-a is OK: OK - load average: 1.01, 1.22, 1.46 [09:31:21] SMF on z-dat-s7-a is OK: OK - all services online [09:31:22] SMF on z-dat-s4-a is OK: OK - all services online [09:31:28] SMTP on z-dat-s3-a is OK: SMTP OK - 0.003 sec. response time [09:31:39] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [09:31:39] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:31:39] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:31:39] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:34:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:47:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 423828 MB (7% inode=41%): [09:57:21] Merlissimo: The "possibly died people" page in es.wiki hasn't been updated by MerlBot for 6 days, are you aware of this? [09:57:55] jem-: yes, replag on toolserver is to high [09:59:08] @replag [09:59:09] jem-: s2-user: 1w 20h 29m 30s [-0.19 s/s]; s3-rr-a: 20s [-0.26 s/s]; s3-user: 20s [-0.26 s/s]; s6-rr-a: 1d 22h 17m 25s [+0.83 s/s]; s6-user: 1d 22h 17m 25s [+0.83 s/s] [09:59:25] I see [09:59:28] s2 and s6 [09:59:38] /sql on cassia is WARNING: DISK WARNING - free space: /sql 92778 MB (7% inode=99%): [09:59:44] 3(assigned) [TS-1404] View commonswiki_p.globalimagelinks is missing on cassia <10https://jira.toolserver.org/browse/TS-1404> (Marlen Caemmerer) [10:00:31] Let's hope it's fixed soon [10:01:43] 3(commented) [TS-1404] View commonswiki_p.globalimagelinks is missing on cassia <10https://jira.toolserver.org/browse/TS-1404> (Marlen Caemmerer) [10:01:45] i think s6 replag will increase for some days now [10:02:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:03:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:04:48] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41919 MB (10% inode=99%): [10:06:43] 3(assigned) [TS-1404] View commonswiki_p.globalimagelinks is missing on cassia <10https://jira.toolserver.org/browse/TS-1404> (Marlen Caemmerer) [10:06:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:09:08] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 166637 [10:12:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:12:48] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 66957 MB (16% inode=99%): [10:14:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:34:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:47:19] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 425888 MB (8% inode=41%): [11:03:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:03:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:06:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:09:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 167003 [11:12:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:14:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:35:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:35:49] 3(commented) [TS-1404] View commonswiki_p.globalimagelinks is missing on cassia <10https://jira.toolserver.org/browse/TS-1404> (DaB.) [11:47:28] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 419714 MB (7% inode=40%): [12:03:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:03:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:06:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:07:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:10:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 159778 [12:13:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:14:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:22:08] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:36:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:47:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 414880 MB (7% inode=40%): [13:03:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:04:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:06:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:11:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 146710 [13:13:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:14:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:36:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:36:39] [[Special:Log/newusers]] create 10 * RepairToolbox * (New user account) [13:47:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 411941 MB (7% inode=40%): [13:56:28] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198655 MB (20% inode=99%): [13:56:58] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42396 MB (10% inode=99%): [14:04:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:04:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:05:59] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 71689 MB (17% inode=99%): [14:07:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:11:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107970 [14:13:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:15:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:17:18] @replag [14:17:19] DaBPunkt: s2-user: 1w 21h 27m 11s [+0.18 s/s]; s3-rr-a: 17s [+0.00 s/s]; s3-user: 17s [+0.00 s/s]; s6-rr-a: 1d 5h 27m 11s [-10.41 s/s]; s6-user: 1d 5h 27m 11s [-10.41 s/s] [14:36:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:47:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 426449 MB (8% inode=41%): [14:54:58] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40333 MB (9% inode=99%): [15:04:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:04:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:07:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:07:58] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23788 MB (5% inode=99%): [15:09:58] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58671 MB (14% inode=99%): [15:11:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89444 [15:13:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:15:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:17:58] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41086 MB (10% inode=99%): [15:31:59] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23488 MB (5% inode=99%): [15:37:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:42:30] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 434206 MB (8% inode=41%): [15:47:58] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36826 MB (9% inode=99%): [15:52:02] Marlen Caemmerer * Re: [Toolserver-l] Anoter SGE question [15:54:59] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23946 MB (5% inode=99%): [15:57:58] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 61290 MB (15% inode=99%): [16:03:58] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 427 MB (18% inode=99%): [16:04:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:04:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:04:58] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 605 MB (23% inode=99%): [16:07:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:11:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 69815 [16:11:59] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43357 MB (10% inode=99%): [16:13:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:15:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:17:58] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 73224 MB (18% inode=99%): [16:27:44] 3(deleted) [ACCAPP-278] Account creation regarding (Marlen Caemmerer) [16:28:30] what is Marlen's IRC nick? [16:31:43] 3(resolved) [TS-1289] Give user sgeadmin acces to sql-mapnik <10https://jira.toolserver.org/browse/TS-1289> (Marlen Caemmerer) [16:32:17] it's a non.trivial mapping [16:32:22] I don't remember it right now [16:32:29] but none of the nicks on this channel ring me a bell [16:37:08] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:37:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:45:59] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42049 MB (10% inode=99%): [16:47:39] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 423373 MB (7% inode=41%): [16:57:58] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23372 MB (5% inode=99%): [17:04:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:05:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:05:58] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53672 MB (13% inode=99%): [17:07:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:11:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30378 [17:14:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:15:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:17:08] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36243 MB (8% inode=99%): [17:21:08] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 68086 MB (16% inode=99%): [17:38:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:39:08] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 34278 MB (8% inode=99%): [17:42:08] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 21298 MB (5% inode=99%): [17:47:39] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 426001 MB (8% inode=41%): [17:49:08] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38514 MB (9% inode=99%): [17:54:46] [[Special:Log/newusers]] create 10 * FroncekBischoff873 * (New user account) [18:05:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:05:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:07:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:12:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18214 [18:13:09] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37812 MB (9% inode=99%): [18:15:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:15:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:18:09] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 21229 MB (5% inode=99%): [18:25:08] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43229 MB (10% inode=99%): [18:32:09] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23570 MB (5% inode=99%): [18:38:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:39:09] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 31971 MB (7% inode=99%): [18:48:38] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 420163 MB (7% inode=40%): [18:59:08] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 393 MB (0% inode=98%): [19:01:09] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 75847 MB (18% inode=99%): [19:05:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:05:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:07:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:11:10] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 34393 MB (8% inode=99%): [19:12:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3814 [19:14:08] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22846 MB (5% inode=99%): [19:14:18] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3557 [19:15:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:15:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:19:51] [[Special:Log/newusers]] create 10 * Erosen * (New user account) [19:20:46] 3(created) [TS-1410] Add me (user ireas) to the tomcat group; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1410> (Robin Krahl) [19:24:08] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 65995 MB (16% inode=99%): [19:27:19] MySQL slave on z-dat-s6-a is OK: Uptime: 887489 Threads: 13 Questions: 173452214 Slow queries: 75712 Opens: 1439620 Flush tables: 1 Open tables: 2853 Queries per second avg: 195.441 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1729 [19:38:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:39:08] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41545 MB (10% inode=99%): [19:48:38] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 420377 MB (7% inode=40%): [19:52:09] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24209 MB (5% inode=99%): [19:57:04] Betacommand: nosy [19:57:17] Dispenser: thanks [19:57:26] * Betacommand notes that he never sees that nick [20:03:11] Betacommand: nosy [20:03:31] She did an interview http://toolserver.org/~bryan/TsLogBot/wikimedia-toolserver_2011-10-27.txt starting at 19:00 UTC [20:03:54] * Dispenser mutters something about the up key [20:04:14] Dispenser: thats what I thought, I just remember that river had about a dozen different nicks, and wasnt sure if he was just missing one [20:05:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:05:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:07:08] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 33636 MB (8% inode=99%): [20:08:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:15:18] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23178 MB (5% inode=99%): [20:15:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:16:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:27:19] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38635 MB (9% inode=99%): [20:31:18] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 73562 MB (18% inode=99%): [20:32:28] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:07] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:38:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:38:59] DaBPunkt: can you rm /opt/zeus/web/log/* on willow (the login server)? [20:41:25] removed the hole zeus-dir [20:42:42] Yay! [20:43:49] DaBPunkt: any reason that we cannot login via the API on the toolserver wiki> [20:48:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410994 MB (7% inode=40%): [21:05:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:05:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:08:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:14:45] 3(created) [TS-1411] TS wiki API login issues; Toolserver: Wiki; Bug <10https://jira.toolserver.org/browse/TS-1411> (Betacommand) [21:15:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:16:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:24:10] no intendedly ones. fill in a bugreport [21:38:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:49:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410898 MB (7% inode=40%): [22:00:18] /sql on cassia is WARNING: DISK WARNING - free space: /sql 93738 MB (7% inode=99%): [22:05:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:05:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:08:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:15:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:16:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:31:10] nacht ts [22:39:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:49:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410796 MB (7% inode=40%): [23:02:18] is there someone with experience using a MMP? I have a problem with cron (it doesn't work) [23:02:57] apper: cron is dead, long live cronie [23:04:41] I'm using crontab/cron on willow on my user account and it works... [23:04:58] Maybe you're lucky [23:05:04] hmm... okay [23:05:12] I'll have a look at cronie [23:05:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:06:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:08:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:08:43] Dispenser: cronie works, thanks a lot [23:16:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:17:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:40:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:50:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413451 MB (7% inode=40%):