[00:03:23] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:06:53] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35022 MB (6% inode=99%): [00:48:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:49:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 186109.000000 [00:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 239969.000000 [00:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112894.000000 [00:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:50:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 399192 MB (7% inode=64%): [00:50:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:50:23] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [00:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:50:23] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89079.000000 [00:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:50:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:50:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:50:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:50:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:50:44] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 292759.000000 [00:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72104 MB (11% inode=99%): [00:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:51:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 224587.000000 [00:51:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 232764.000000 [00:51:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:51:53] / on thyme is UNKNOWN: NRPE: Unable to read output [00:51:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89048 [00:52:53] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:53:53] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35135 MB (6% inode=99%): [00:59:22] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [00:59:22] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [01:10:23] MySQL on z-dat-s7-a is OK: Uptime: 1031 Threads: 3 Questions: 2152 Slow queries: 4 Opens: 160 Flush tables: 1 Open tables: 149 Queries per second avg: 2.87 [01:10:23] MySQL slave on z-dat-s7-a is OK: Uptime: 1031 Threads: 3 Questions: 2153 Slow queries: 4 Opens: 160 Flush tables: 1 Open tables: 149 Queries per second avg: 2.88 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1032 [01:11:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 30041 MB (3% inode=99%): [01:48:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 187932.000000 [01:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 242330.000000 [01:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112842.000000 [01:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 398948 MB (7% inode=64%): [01:50:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:50:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:50:22] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [01:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:50:23] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:50:32] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:50:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85325.000000 [01:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:50:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:50:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 293236.000000 [01:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71845 MB (11% inode=99%): [01:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 236330.000000 [01:51:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 226195.000000 [01:51:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:51:53] / on thyme is UNKNOWN: NRPE: Unable to read output [01:51:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85289 [01:52:53] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:54:53] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35134 MB (6% inode=99%): [01:59:53] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [01:59:53] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [02:12:53] MySQL slave on z-dat-s3-a is OK: Uptime: 1092 Threads: 9 Questions: 81 Slow queries: 0 Opens: 63 Flush tables: 1 Open tables: 55 Queries per second avg: 0.74 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1213 [02:12:53] MySQL on z-dat-s3-a is OK: Uptime: 1092 Threads: 9 Questions: 83 Slow queries: 0 Opens: 63 Flush tables: 1 Open tables: 55 Queries per second avg: 0.76 [02:19:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:48:33] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2133.000000 [02:48:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 185782.000000 [02:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 244422.000000 [02:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113424.000000 [02:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 398749 MB (7% inode=64%): [02:50:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:50:22] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [02:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:50:23] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:50:32] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [02:50:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81103.000000 [02:50:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:50:32] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:50:44] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 293767.000000 [02:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71575 MB (11% inode=99%): [02:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:51:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 239931.000000 [02:51:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 228047.000000 [02:51:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:51:53] / on thyme is UNKNOWN: NRPE: Unable to read output [02:51:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 81008 [02:54:33] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1675.000000 [03:19:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:48:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:49:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [03:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 185697.000000 [03:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 246625.000000 [03:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114291.000000 [03:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 398814 MB (7% inode=64%): [03:50:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:50:17] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:50:17] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:50:22] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [03:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [03:50:23] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:50:32] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:50:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74389.000000 [03:50:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:50:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:50:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 294741.000000 [03:50:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71058 MB (11% inode=99%): [03:50:54] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:50:54] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:51:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 243532.000000 [03:51:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 230083.000000 [03:51:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:51:53] / on thyme is UNKNOWN: NRPE: Unable to read output [03:51:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 74298 [04:19:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:48:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [04:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 187676.000000 [04:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 248949.000000 [04:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115668.000000 [04:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:02] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:50:02] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:50:02] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 398492 MB (7% inode=64%): [04:50:12] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:50:25] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [04:50:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:50:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:50:25] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [04:50:33] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72750.000000 [04:50:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:50:34] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:45] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:50:45] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 296096.000000 [04:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70791 MB (11% inode=99%): [04:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 247132.000000 [04:51:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 232234.000000 [04:51:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:51:54] / on thyme is UNKNOWN: NRPE: Unable to read output [04:51:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 72767 [04:54:13] Free Memory on damiana is WARNING: WARNING - 5.4% (449364 kB) free! [04:56:13] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (411856 kB) free! [05:19:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:45:33] Oh please, how much until toolserver is back? T.T [05:48:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:49:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 189293.000000 [05:49:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 251277.000000 [05:49:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116235.000000 [05:49:54] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 398431 MB (7% inode=64%): [05:50:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:50:23] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [05:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:50:33] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [05:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69279.000000 [05:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70492 MB (11% inode=99%): [05:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:44] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:51:44] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 297473.000000 [05:51:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 250732.000000 [05:51:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 234476.000000 [05:51:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:51:54] / on thyme is UNKNOWN: NRPE: Unable to read output [05:51:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 69117 [05:56:13] Free Memory on damiana is CRITICAL: CRITICAL - 2.2% (182092 kB) free! [06:19:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:48:54] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 191737.000000 [06:49:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 254243.000000 [06:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 398352 MB (7% inode=64%): [06:50:13] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118344.000000 [06:50:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:50:22] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [06:50:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:50:33] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:50:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61985.000000 [06:50:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70231 MB (11% inode=99%): [06:50:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:51:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:51:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:44] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:52:44] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 300027.000000 [06:52:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 254392.000000 [06:52:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 236365.000000 [06:52:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:52:54] / on thyme is UNKNOWN: NRPE: Unable to read output [06:52:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61873 [06:57:13] Free Memory on damiana is CRITICAL: CRITICAL - 2.0% (164980 kB) free! [07:14:33] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [07:19:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:33:16] Marlen Caemmerer * Re: [Toolserver-l] Ipv6 issues (fwd) [07:38:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [07:48:54] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:14] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 195337.000000 [07:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 257843.000000 [07:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 398038 MB (7% inode=64%): [07:50:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121944.000000 [07:50:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:50:22] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [07:50:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:50:24] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:50:33] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56437.000000 [07:50:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69887 MB (11% inode=99%): [07:50:54] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:51:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:51:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 257992.000000 [07:53:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:53:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 303687.000000 [07:53:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:53:53] / on thyme is UNKNOWN: NRPE: Unable to read output [07:53:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56289 [07:57:13] Free Memory on damiana is CRITICAL: CRITICAL - 1.8% (152044 kB) free! [07:57:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 234836.000000 [08:19:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:38:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7256.000000 [08:48:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:49:28] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 198937.000000 [08:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 261443.000000 [08:50:02] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:50:02] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 397941 MB (7% inode=64%): [08:50:13] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 125544.000000 [08:50:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:50:23] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [08:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:50:33] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54499.000000 [08:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69565 MB (11% inode=99%): [08:50:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:52] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:51:54] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 261591.000000 [08:53:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:53:44] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 307287.000000 [08:53:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:53:53] / on thyme is UNKNOWN: NRPE: Unable to read output [08:53:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54289 [08:57:12] Free Memory on damiana is CRITICAL: CRITICAL - 2.0% (165920 kB) free! [08:57:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 233657.000000 [09:19:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:38:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10856.000000 [09:48:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:49:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 202538.000000 [09:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 265043.000000 [09:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 397880 MB (7% inode=64%): [09:50:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:50:13] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 129144.000000 [09:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:50:23] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [09:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:50:33] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50853.000000 [09:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69222 MB (11% inode=99%): [09:50:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:51:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 265192.000000 [09:53:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:53:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 310888.000000 [09:53:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:53:53] / on thyme is UNKNOWN: NRPE: Unable to read output [09:53:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50699 [09:57:13] Free Memory on damiana is CRITICAL: CRITICAL - 1.9% (155096 kB) free! [09:57:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 232583.000000 [10:19:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:27:22] Load avg. on thyme is CRITICAL: (Service Check Timed Out) [10:27:30] toolserver.org HTTP on ortelius is CRITICAL: (Service Check Timed Out) [10:27:31] Sun Grid Engine execd on ortelius is CRITICAL: (Service Check Timed Out) [10:27:32] /sql on ptolemy is CRITICAL: (Service Check Timed Out) [10:27:32] RAID on thyme is CRITICAL: (Service Check Timed Out) [10:28:51] uh oh [10:30:56] * DarkoNeko goes read the ML [10:47:25] it looks like verything's down, crons aren't running, ssh attempt apparently ignore ssh lkeys and ask for password. That one might be related to the IP chang ething tho [11:43:52] we dead again? [12:00:01] yes [12:06:13] I can access via ssh now. [12:42:03] oh ?! [12:42:06] i'll try [12:42:58] \o/ works for me too [12:43:01] now to check my scripts [12:59:18] how long does it usually take to get approved for TS? [13:01:44] greenrosetta: depends [13:02:01] it can take as little as a few hours or several weels [13:02:09] *weeks [13:02:34] Is there one person in charge, or is this done by votes? [13:08:06] greenrosetta: Usually DaB. is in charge, and there is no voting, a reasonable petition is usually enough [13:10:17] Betacommand: Could I take a peek at that webpage & assocatioed python code you sent me a link to yesterday? I want to examine stuff while I wait [13:30:25] greenrosetta: just use php if you know it [13:30:43] I've been meaing to learn python [13:31:36] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:31:36] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:31:36] SMTP on adenia is OK: SMTP OK - 0.011 sec. response time [13:31:36] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [13:32:56] Free Memory on damiana is OK: OK - 91.8% (7693420 kB) free. [13:34:47] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31864 MB (3% inode=99%): [13:35:46] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:35:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 216124.000000 [13:36:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19722.000000 [13:36:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 324238.000000 [13:36:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19575 [13:37:37] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:37:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:37:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:40:14] Betacommand: Just got a simple hello-world running in py using Flask. that was pretty easy [13:40:29] didnt even have to install apache [13:41:00] ssh connecting works, but it seems that at least willow doesn't have access to the outside internet [13:43:04] dns wise [13:43:31] DarkoNeko, its ip6 routes fail [13:43:48] if you use ipv4 it works [13:43:55] hmm [13:44:27] / on thyme is UNKNOWN: NRPE: Unable to read output [13:44:29] traceroute: Warning: wikimedia.org has multiple addresses; using 2620:0:860:ed1a:: | traceroute: Warning: Multiple interfaces found; using 2a02:ec80:101::2:4 @ bnx0:1 [13:44:32] ah, figures [13:47:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35016 MB (6% inode=99%): [13:47:29] DarkoNeko, use traceroute -A inet wikimedia.org [13:48:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:48:20] that one (ipv4) works [13:48:22] thanks [13:48:27] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68929 MB (11% inode=99%): [13:48:46] I suppose that scripts run on the server try the ipv6 by default tho [13:49:37] ipv6 is used in preference to ipv4, yes [13:50:02] it seems that writing "DEFAULT_IP=IP_VERSION4" into /etc/default/inet_type [13:50:10] supposedly disables ipv6 in solaris [13:50:23] but that would have to be done by nosy or DaB [13:51:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 398145 MB (7% inode=64%): [13:52:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:52:46] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:53:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:55:36] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:55:57] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:58:48] grmbl [14:01:57] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:01:57] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [14:02:06] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:02:06] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:02:06] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:02:06] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:27] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:27] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:06:29] does toolserver support any specific python frameworks? [14:13:15] Platonides * Re: [Toolserver-l] Ipv6 issues (fwd) [14:29:37] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [14:29:46] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [14:31:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:31:07] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [14:31:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8170.000000 [14:31:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28408.000000 [14:31:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 281934.000000 [14:31:16] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 281895.000000 [14:31:26] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 146019.000000 [14:31:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:31:26] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:31:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:31:36] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:31:37] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:31:46] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:31:57] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:31:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:07] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [14:35:47] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:35:57] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 219723.000000 [14:36:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12370.000000 [14:36:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 327837.000000 [14:36:26] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12298 [14:36:46] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2198 [14:37:37] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:37:47] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2191 [14:38:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:38:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:44:27] / on thyme is UNKNOWN: NRPE: Unable to read output [14:46:36] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2067 [14:49:17] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:49:27] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68127 MB (11% inode=99%): [14:49:59] http://toolserver.org/~hersfold/newfakeSULutil.php?username=SyriaAram failed to connect on Cluster 6. :( [14:51:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 397751 MB (7% inode=64%): [14:52:47] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:53:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:53:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:55:32] http://toolserver.org/~quentinv57/tools/sulinfo.php?showinactivity=1&showblocks=1&username=SyriaAram says "Warning : The SQL server s6 is down or having issues. Consequently, the following wikis won't be displayed : ruwiki, jawiki and frwiki." and "Due to an issue on Toolserver's s7 databases server, these informations cannot been displayed. Please wait that the problem is fixed [14:55:33] or request help at #wikimedia-toolserver." [14:55:37] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:55:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:00:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3619 [15:01:56] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:01:57] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [15:02:06] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:06] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:02:06] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:02:06] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:02:27] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:02:27] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:46] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3619 [15:07:36] MySQL on z-dat-s6-a is OK: Uptime: 2642 Threads: 16 Questions: 1205 Slow queries: 1 Opens: 89 Flush tables: 1 Open tables: 81 Queries per second avg: 0.456 [15:15:00] Problem resolved. [15:22:46] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3481 [15:26:36] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3639 [15:31:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:31:07] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [15:31:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11770.000000 [15:31:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32009.000000 [15:31:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 285534.000000 [15:31:17] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 285494.000000 [15:31:57] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:31:57] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:31:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:06] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 233603.000000 [15:32:26] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 149679.000000 [15:32:26] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:32:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [15:32:36] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:32:36] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:33:46] MySQL slave on z-dat-s6-a is OK: Uptime: 4210 Threads: 18 Questions: 415353 Slow queries: 144 Opens: 264 Flush tables: 1 Open tables: 238 Queries per second avg: 98.658 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1678 [15:36:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 331437.000000 [15:36:46] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:36:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 223383.000000 [15:37:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6343.000000 [15:37:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6327 [15:38:37] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:38:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:38:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:44:27] / on thyme is UNKNOWN: NRPE: Unable to read output [15:49:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:49:27] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67841 MB (11% inode=99%): [15:51:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 397273 MB (7% inode=64%): [15:52:27] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:53:46] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:54:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:54:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:55:37] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:56:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:58:06] RAID on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:16] / on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:16] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:16] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:16] /sql on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:16] / on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:17] / on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:17] / on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:27] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:27] /tmp on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:27] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:27] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:36] / on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:36] /tmp on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:36] /tmp on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:36] Load avg. on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:36] Load avg. on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:37] /tmp on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:37] Load avg. on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:38] /tmp on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:38] Load avg. on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:46] Environment IPMI on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:46] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [15:58:57] Load avg. on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:57] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [15:59:06] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:59:06] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:59:17] / on z-dat-s4-a is OK: DISK OK - free space: / 6641 MB (22% inode=84%): [15:59:17] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:59:17] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 76767 MB (12% inode=99%): [15:59:17] / on z-dat-s3-a is OK: DISK OK - free space: / 6641 MB (22% inode=84%): [15:59:17] / on z-dat-s6-a is OK: DISK OK - free space: / 6641 MB (22% inode=84%): [15:59:17] / on z-dat-s7-a is OK: DISK OK - free space: / 6641 MB (22% inode=84%): [15:59:27] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [15:59:27] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 45025 MB (99% inode=97%): [15:59:27] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 283353 MB (29% inode=99%): [15:59:27] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 368047 MB (45% inode=99%): [15:59:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 34815 MB (6% inode=99%): [15:59:28] / on hyacinth is OK: DISK OK - free space: / 6641 MB (22% inode=84%): [15:59:28] /tmp on hyacinth is OK: DISK OK - free space: /tmp 47309 MB (99% inode=97%): [15:59:36] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [15:59:36] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 47288 MB (99% inode=97%): [15:59:36] Load avg. on z-dat-s6-a is OK: OK - load average: 0.79, 0.32, 0.38 [15:59:36] Load avg. on z-dat-s4-a is OK: OK - load average: 0.79, 0.32, 0.38 [15:59:36] Load avg. on z-dat-s7-a is OK: OK - load average: 0.79, 0.32, 0.38 [15:59:37] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 47288 MB (99% inode=97%): [15:59:37] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 47289 MB (99% inode=97%): [15:59:46] Load avg. on z-dat-s3-a is OK: OK - load average: 0.81, 0.33, 0.38 [15:59:46] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [15:59:46] Load avg. on hyacinth is OK: OK - load average: 0.86, 0.35, 0.39 [16:00:06] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:01:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:01:37] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [16:02:06] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:06] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:06] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:06] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:26] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:27] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:57] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:57] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [16:03:46] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [16:05:36] MySQL on z-dat-s7-a is OK: Uptime: 374 Threads: 3 Questions: 600 Slow queries: 1 Opens: 87 Flush tables: 1 Open tables: 78 Queries per second avg: 1.604 [16:06:46] MySQL on z-dat-s6-a is OK: Uptime: 449 Threads: 4 Questions: 1320 Slow queries: 2 Opens: 77 Flush tables: 1 Open tables: 66 Queries per second avg: 2.939 [16:06:56] MySQL slave on z-dat-s6-a is OK: Uptime: 457 Threads: 4 Questions: 3891 Slow queries: 4 Opens: 105 Flush tables: 1 Open tables: 90 Queries per second avg: 8.514 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1062 [16:07:05] @replag [16:07:36] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3525 [16:11:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3575.000000 [16:11:27] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3555 [16:11:37] MySQL slave on z-dat-s4-a is OK: Uptime: 179510 Threads: 1 Questions: 15905351 Slow queries: 7 Opens: 93 Flush tables: 1 Open tables: 69 Queries per second avg: 88.604 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1588 [16:16:26] MySQL on z-dat-s3-a is OK: Uptime: 1025 Threads: 14 Questions: 3883 Slow queries: 1 Opens: 224 Flush tables: 1 Open tables: 206 Queries per second avg: 3.788 [16:17:47] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3572 [16:20:04] @replag [16:22:46] MySQL slave on z-dat-s7-a is OK: Uptime: 1404 Threads: 17 Questions: 764078 Slow queries: 16 Opens: 604 Flush tables: 1 Open tables: 592 Queries per second avg: 544.215 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1524 [16:31:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:31:07] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [16:31:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15370.000000 [16:31:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35609.000000 [16:31:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:27] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 289104.000000 [16:31:51] some random revisions are missing from replicated database on toolserver (around 0.1%) what is the reason for this (and is it intentional or unintentional?) [16:32:07] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 237203.000000 [16:32:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 289194.000000 [16:32:27] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 153279.000000 [16:32:27] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:32:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:32:37] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:32:37] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:32:46] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:32:57] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:32:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 335037.000000 [16:36:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:47] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:36:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 226984.000000 [16:38:07] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1688.000000 [16:38:27] MySQL slave on rosemary is OK: Uptime: 954848 Threads: 9 Questions: 849071252 Slow queries: 289146 Opens: 43704 Flush tables: 1 Open tables: 3386 Queries per second avg: 889.221 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1606 [16:38:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:38:50] anyone alive? :) [16:38:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:38:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:44:26] / on thyme is UNKNOWN: NRPE: Unable to read output [16:49:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:50:27] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67507 MB (11% inode=99%): [16:52:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 396904 MB (7% inode=64%): [16:53:36] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3412 [16:53:47] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:54:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:54:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:55:38] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:56:57] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:00:36] MySQL slave on z-dat-s3-a is OK: Uptime: 3675 Threads: 16 Questions: 2348566 Slow queries: 101 Opens: 16038 Flush tables: 1 Open tables: 16020 Queries per second avg: 639.65 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1728 [17:02:07] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:07] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:02:07] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:02:07] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:02:27] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:02:27] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:56] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:03:56] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [17:31:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:31:27] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 292705.000000 [17:32:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:32:06] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [17:32:06] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19031.000000 [17:32:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39269.000000 [17:32:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 292794.000000 [17:32:27] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156879.000000 [17:32:27] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:32:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [17:32:36] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:32:36] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:32:46] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:32:57] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:32:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:09] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 240862.000000 [17:36:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 338665.000000 [17:36:47] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:37:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 230643.000000 [17:39:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:39:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:39:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:45:27] / on thyme is UNKNOWN: NRPE: Unable to read output [17:49:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:51:27] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67269 MB (11% inode=99%): [17:52:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 396540 MB (7% inode=64%): [17:54:46] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:55:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:55:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:56:37] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:57:57] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:02:06] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:06] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:02:07] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:02:07] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:02:26] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:02:26] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:36] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2154 [18:04:46] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2181 [18:04:57] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:04:57] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [18:07:56] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2136 [18:16:46] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:16:56] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [18:19:37] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1981 [18:21:56] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2963 [18:22:37] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:26:05] hello [18:26:25] i will reboot willow in about half an hour - 1900 UTC [18:29:09] nosy: hello, what's wrong with it? [18:29:37] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3634 [18:30:36] MySQL on z-dat-s7-a is OK: Uptime: 858 Threads: 1 Questions: 50 Slow queries: 0 Opens: 30 Flush tables: 1 Open tables: 23 Queries per second avg: 0.58 [18:31:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:31:27] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 296305.000000 [18:32:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:32:06] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [18:32:06] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22631.000000 [18:32:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42869.000000 [18:32:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:46] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3599 [18:32:46] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:33:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 296454.000000 [18:33:27] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 160538.000000 [18:33:27] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:33:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [18:33:36] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:33:37] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:33:56] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:33:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:06] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 244523.000000 [18:36:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:46] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 342265.000000 [18:37:36] MySQL slave on z-dat-s4-a is OK: Uptime: 188270 Threads: 1 Questions: 16438312 Slow queries: 7 Opens: 93 Flush tables: 1 Open tables: 69 Queries per second avg: 87.312 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1753 [18:37:46] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:37:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 234243.000000 [18:39:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:39:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:39:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:41:36] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3516 [18:43:46] MySQL slave on z-dat-s7-a is OK: Uptime: 1648 Threads: 4 Questions: 555571 Slow queries: 24 Opens: 592 Flush tables: 1 Open tables: 584 Queries per second avg: 337.118 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1787 [18:46:26] / on thyme is UNKNOWN: NRPE: Unable to read output [18:49:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:50:27] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67091 MB (10% inode=99%): [18:50:37] anybody around? [18:51:13] Just ask your question. [18:52:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 396190 MB (7% inode=63%): [18:53:19] Susan: Okay [18:53:29] Susan: I can't seem to SSH into tool server [18:53:45] e.g., yarrow.toolserver.org [18:54:36] MySQL slave on z-dat-s3-a is OK: Uptime: 10517 Threads: 21 Questions: 5220786 Slow queries: 5706 Opens: 24535 Flush tables: 1 Open tables: 16384 Queries per second avg: 496.413 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1703 [18:54:47] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:55:05] ssh to willow works for me. [18:55:15] From willow, I can ssh into yarrow. [18:55:33] ssh directly to yarrow from my local machine also works. [18:55:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:55:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:55:38] preilly: What error are you getting? [18:55:45] The IP address recently changed. [18:55:47] Permission denied (publickey,password,keyboard-interactive,hostbased). [18:55:53] But SSH should work fine, as long as your account isn't expired. [18:56:07] nosy: ^^ [18:56:36] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:57:03] preilly: your account is not expired, ipv4 login is working via ssh [18:57:06] https://toolserver.org/~preilly/ your account seems fine. The SSH log will know the answer, I imagine. [18:57:13] we are currently having issues with ipv6 [18:57:30] Does SSH use IPv6? [18:57:36] you might go this path - you should see in ssh -v which address is used [18:57:39] nosy: I think I might have the wrong key on keyserver [18:57:54] That would also do it. ;-) [18:57:56] Susan: yes indeed solaris prefers ipv6 over v4 [18:57:57] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:58:08] If you need a new key added, I think you can file a ticket in JIRA (jira.toolserver.org). [18:58:13] nosy: Solaris. :-/ [18:58:13] preilly: please send an email to ts-admin then [18:58:32] Susan: see http://pastebin.mozilla.org/2282326 [18:58:36] Susan: you want to help migrate to linux? [18:58:41] ts-admins@toolserver.org [18:58:47] feel welcome ;) [18:59:03] its going slow - lack of personal and system ressources [18:59:15] Looks like it's using IPv4. [19:00:25] preilly: So as nosy said, you'll want to e-mail a new public key to ts-admins@toolserver.org. [19:02:43] Susan: I emailed them [19:02:48] Cool. [19:03:35] ok, willow seems to really work after reboot [19:03:41] can anyone confirm please? [19:03:49] i can ping ipv6 and v4 [19:03:57] * Susan looks. [19:04:04] ssh into willow works. [19:04:15] mzmcbride@willow:~$ ping google.com [19:04:15] google.com is alive [19:04:20] That looks good. [19:04:25] nosy: Can you tell me the public key that I've got on TS now? [19:04:29] @replag [19:04:36] tsbot seems broken. [19:04:45] I think it was related to the IPv6 issues. [19:04:47] Could it be rebooted? [19:04:51] preilly: patrick.reilly@gmail.com [19:05:06] you really want tsbot to spam? ;) [19:05:19] tsbot is the replag bot, isn't it? [19:05:22] tsnag is the one I ignore. ;-) [19:05:24] I think. [19:06:03] true [19:06:34] preilly: how does the corresponding public key end on your side? [19:06:52] here it ends with 7VIgmapomPvWmzmvZB53rEo1TVGlSet patrick.reilly@gmail.com [19:07:23] nosy: yeah that's the right key [19:07:34] nosy: and that is the one installed? [19:07:51] preilly please wait a little i need to look currently for other things too [19:08:14] nosy: okay no worries… I appreciate the help [19:08:27] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:08:27] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:41] @replag [19:08:56] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:08:57] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [19:09:06] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:06] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:09:06] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:09:06] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:17:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 299097.000000 [19:17:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 247116.000000 [19:17:26] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 344701.000000 [19:17:26] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:17:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:17:30] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:17:30] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:17:30] / on thyme is UNKNOWN: NRPE: Unable to read output [19:17:39] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:17:40] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:18:01] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:18:11] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:25:08] nosy: both nightshade and willow work for me in v4 and v6 [19:25:20] Coren: thanks for looking :) [19:25:25] for me too [19:25:29] even from outside [19:25:50] i am happy if people tell me things work again that did not before [19:25:56] i only did a reboot [19:26:48] it seems nothing has changed only the multiple ipv6 interfaces with the same address on each is gone [19:27:52] preilly: as of your ssh [19:28:01] it seems you already use the right key [19:28:06] i get strange error messages [19:28:13] ah ill send you email about that [19:29:01] nosy: okay [19:29:31] nosy: thanks for looking [19:34:27] Betacommand: I've finished my front end. I used python with Flask. Any chance you could show me one of your python scripts so I can figure out how to add the DB stuff? [19:34:42] @replag [19:40:06] greenrosetta: one sec [19:40:20] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.010 second response time [19:40:48] greenrosetta: https://wiki.toolserver.org/view/Database_access#Python [19:41:11] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.249 second response time [19:41:29] @replag all [19:42:58] tsbot: ? [19:47:27] Betacommand: thx. What's DAB's en-wiki name? [19:47:43] greenrosetta: just wait [19:47:52] no, I just wanted to see his page [19:47:55] greenrosetta: when dab gets to it he gets to it [19:48:12] greenrosetta: he really isnt on enwiki [19:48:17] oic [19:48:48] Betacommand: Are you familiar with Flask? [19:48:55] no [19:49:09] its a front end for using python, but doesnt require apache [19:49:59] nice little package. Once I got it configred it was pretty easy to build the pages and keep the buisness logic seperate [19:54:44] tsbot: having a bad day? [19:54:54] it always goes when someone queries [19:55:21] @replag [19:55:24] nosy: been having issues with IRC bots since the maintenance [20:01:59] Betacommand: yeah the god damn maintenance [20:02:04] :D [20:02:10] what a mess [20:02:24] nosy: half my bots are still down :( [20:02:36] Betacommand: whats broken? [20:03:08] do you have an example? [20:03:18] I still cant get any of my IRC bots up [20:03:27] which host? [20:03:38] and which command? [20:03:47] Nightshade or willow [20:04:01] betacommand@nightshade:~/Watchbot$ screen php persist.php [20:05:06] and now its comming up on williow [20:05:34] Betacommand: works on willow now? [20:05:41] yeah [20:07:11] root@nightshade:/home/rnosy# telnet !$ 6667 [20:07:11] telnet irc.freenode.net 6667 [20:07:11] Trying 2a02:2f0d:bff0:0:81:18:85:57... [20:07:11] Connected to chat.freenode.net. [20:07:11] Escape character is '^]'. [20:07:11] :hobana.freenode.net NOTICE * :*** Looking up your hostname... [20:07:11] :hobana.freenode.net NOTICE * :*** Checking Ident [20:07:12] :hobana.freenode.net NOTICE * :*** Found your hostname [20:07:12] :hobana.freenode.net NOTICE * :*** Got Ident response [20:07:52] Betacommand: i assume the connection to freenode should work from nightschade [20:08:42] nosy: http://pastebin.com/5D9SPZ7p [20:08:47] example of output [20:10:53] Betacommand: where does the exit error code come from? [20:11:12] is it php, a php function or some external program? [20:13:24] nosy: termination of irc.php [20:15:19] Betacommand: can you tell why? [20:15:23] why it ends? [20:16:38] looks like either a connection to freenode or irc.wikimedia failed [20:16:41] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 393350 MB (7% inode=63%): [20:16:41] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:16:42] Betacommand: do you use windows? I'm trying to install oursql [20:16:48] having some issues [20:16:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:16:50] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:16:50] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:16:55] @replag [20:17:00] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [20:17:00] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 240187.000000 [20:17:00] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:01] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:17:01] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:17:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:17:01] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:17:11] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:17:12] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:17:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:17:12] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28934.000000 [20:17:12] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:17:12] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:12] nosy: My feeling is that reba and tsbot and Betacommand all have the same problem. [20:17:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49173.000000 [20:17:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:24] Something between the Toolserver and freenode. [20:17:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 302660.000000 [20:17:24] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 250718.000000 [20:17:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 302699.000000 [20:17:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:24] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:17:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348302.000000 [20:17:24] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:17:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 166778.000000 [20:17:24] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:24] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66410 MB (10% inode=99%): [20:17:31] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:17:31] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:17:36] Susan: quite possible but I need a testing [20:17:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:17:41] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:17:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:17:41] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:17:41] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:17:42] ill try a irc client [20:17:58] Hmm. [20:17:59] Susan: reba might be better suited for nosy's testing [20:18:00] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:18:03] My other IRC bot is working. [20:18:27] hey its me [20:18:30] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:18:30] / on thyme is UNKNOWN: NRPE: Unable to read output [20:18:32] a test from willow [20:18:38] no nightshade [20:18:44] /home/mzmcbride/scripts/irc/reba.py is the script I'm using. [20:18:55] I'm not sure why reba is disconnecting so often lately. [20:19:00] It's a simple e-mail poll script. [20:19:11] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:11] ok ill go [20:19:19] it seems i can use irc from nightshadew [20:19:25] nightshade... [20:19:34] so ill have to look elsewhere [20:25:40] Betacommand: Is there a place to download a sample of the database with some sample data populated? [20:26:07] no [20:26:23] phooey [20:27:18] Where can i find the schema? [20:27:39] it's possible that the toolserver ips had an higher connection limit for toolserver ips [20:27:53] Platonides: thats right [20:27:56] * Platonides realises the redundancy of the above line [20:28:04] thanks for your mail btw [20:28:17] no problem :) [20:28:22] seems a reboot was everything that was needed [20:28:35] Platonides: yes it probably was only the many interfaces [20:28:50] with the same ipv6 ip [20:29:00] it looked very weird [20:29:01] routing is fine as far as i can tell [20:29:18] the local default router is the same as for the old subnet [20:29:35] i guess the back of the address is related to the mac of the router interface [20:29:45] so that might be the same [20:30:02] fe80:: addresses are usually constructed from macs [20:30:24] yes so i assume i get the correct default gw machine there [20:31:00] I had compared it with the gateway for... nightshade, I think [20:31:00] Platonides: unfortunatelly tsbot should not fail since it should still come from the old ranges [20:31:14] yes it does [20:31:48] seeing different gateways seemed a possble error [20:31:50] but telling freenode we have a new range is quite important to do [20:31:57] I don't know how they are connected [20:32:07] I think wmf also had toolserver ips in a whitelist [20:32:28] funny thing is though: i thought and what a network op also told me the default gw fe80:: was enough [20:32:29] although as it was wmf who changed the assigned ips, they might already have changed it [20:33:19] but if i dont have the default gw on the external address i cannot go to the internal services in the 10. ipv4 range [20:33:37] thats something ill ask mark tomorow [20:34:11] i dont trust the idea wmf thought of that ;) [20:34:25] I don't think they would have realised, either :) [20:34:42] but heh, if something fails due to that it's their faulr [20:34:44] *fault [20:35:56] it's strange that not having the gw on the external ip you had problems in the 10. range [20:36:15] I would have expected it to be able to direct through that, being on the same subnet [20:36:26] without an explicit route [20:37:50] I guess the moral is "don't trust default behavior to do the right thing" [20:43:57] Hello all. I'm back [20:44:23] hello DaBPunkt [20:44:51] hi DaBPunkt. how can i modify something in puppet? you write that operator accounts have access, but how to create a password for svn access? [20:45:28] setting ldap password in only possible on solaris where operator accounts have no access [20:45:29] Merlissimo, we have to sent it to him [20:45:35] Merlissimo: create a file in your operator-home with htpasswd -m [20:45:55] I will copy the content to the master-file than [20:47:45] using omerl? [20:48:00] yes [20:48:35] does the realm differ? [20:48:46] I'm not sure if svn will support storing different credentials for the same host [20:48:48] ok i sent an email to the freenode ops telling them our new net ranges [20:49:39] hey DaBPunkt [20:50:48] nosy: why were the multiple logial interfaces not visible for you as root? [20:52:33] i saw but i did not want to send too much output as it might confuse [20:53:11] DaBPunkt: ~omerl/omerl [20:53:13] DaBPunkt, created file at /home/oplatonides/puppet-svn [20:54:51] ok, wait 2 moments [20:55:47] ok. The svn-path is https://svn.toolserver.org/svnroot/system/puppet/ [20:57:45] password wasn't needed for checkout [20:57:58] DaBPunkt/nosy: three questions/requests: [20:58:01] 1) sgeperl (our perl version fix) is unkown on clematis and hawthorn [20:59:03] 2) could you enable renice, so that operator can lower priority instead of killing a task directly? [20:59:52] 3) why is nscd -g (viewing statistics) not possible on debian? could this be enabled? on solaris this is possible [20:59:54] good idea [21:00:47] 1): fixed [21:00:48] Merlissimo, is debian toolserver using nscd ? [21:01:09] it is [21:01:11] 2) Sure, no problem [21:02:09] 3.) I have to look. I realy hate nsd(c) and touch it as few as possible [21:02:17] I guess nscd authors were afraid that some info could be leaked through nscd statistics [21:02:18] yes. i already changed solaris nsswitchand nscd config on my local repository copy. nscd on debian causes sometimes problem because of 100% cpu [21:03:13] DaBPunkt: ich checks statistic on willow, clematis and wolfsbane. but i think checking usage statistics for debian should be done before i change it there, too. [21:04:11] looks like it is just a permission-problem. I will look if I can add it to sudo [21:05:44] I have to leave for a moment, one of my servers hate my tody… [21:09:20] Free Memory on turnera is WARNING: WARNING - 6.4% (539032 kB) free! [21:13:01] ok ill go for today [21:13:04] see you tomorrow [21:15:21] ok, I did a simple commit to the toolserver puppet [21:16:20] I guess what I did would be the prefered way to install packages? [21:16:40] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 392892 MB (7% inode=63%): [21:16:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:16:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:16:50] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:16:50] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:17:10] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:17:11] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:17:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:17:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:12] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32534.000000 [21:17:12] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52772.000000 [21:17:12] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:17:12] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:12] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 306299.000000 [21:17:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 254318.000000 [21:17:22] Free Memory on turnera is OK: OK - 7.1% (594924 kB) free. [21:17:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 306260.000000 [21:17:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:17:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 351902.000000 [21:17:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 170378.000000 [21:17:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:17:24] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66167 MB (10% inode=99%): [21:17:24] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:17:36] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:18:01] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:18:01] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 243847.000000 [21:18:01] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [21:18:01] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:18:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:18:02] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:18:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:18:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:18:30] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:18:30] / on thyme is UNKNOWN: NRPE: Unable to read output [21:18:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:18:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:18:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:18:41] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:18:41] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:19:11] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:21:45] re [21:24:11] Platonides: eee, why are own classes for editor needed? I know that puppet should be small steps, but this small? [21:25:18] well, it was mostly a test [21:25:39] given that there was a class for vim, I supposed it was prefered than an editors class listing several packages [21:25:51] also, I don't know if you have a better way for dmping a list of packages [21:25:55] *dumping [21:26:40] so please advice how you want it to be used :) [21:27:20] Free Memory on turnera is CRITICAL: CRITICAL - 4.1% (345924 kB) free! [21:27:30] seems I already broke the unwritten rule of "Always start the commit msgs with - or #" :P [21:30:03] Platonides: I adopted the puppet-structure of River. I have no problems to change the structure (I'm a puppet-beginner myself) but if we do there should be some kind of guidelines at least for new files – having x styles (x=number of operators) is not really preferd :-) [21:36:47] DaBPunkt, I'm fine with whatever way you like [21:36:57] I'm not a puppet ninja, either [21:38:46] Than let's stcik with rivers style for first. Create only new classes if there is more than 1 thing in it (for example: A package and its config-file) [21:40:48] ok, so how should I have done something like that, of adding a package? [21:40:56] is there a misc class? [21:46:48] yes [21:47:42] misc_user.pp for the userland-servers, misc_std.pp for other servers [21:48:03] I found them now [21:48:12] sorry, I missed them [21:48:18] libmisc_user.pp for the rare case that standalone libery is needed [21:48:40] in fact, both of them were already in ./software/manifests/misc_std.pp [21:48:40] liberies [21:49:00] Platonides: of corse, I use joe as editor myself :) [21:49:13] and nano as fallback [21:49:54] I know they are installed [21:50:58] in fact, it's a bit annoying that the global config file is in /etc/opt/ts/joe/joerc in solaris but in /etc/joe/joerc in debian [21:52:37] yes, but that's the proble for nearly every program [21:52:43] problem [21:53:29] my .joerc contains: :include /etc/opt/ts/joe/joerc [21:53:29] -nobackups [21:53:47] which breaks in debian :P [21:54:00] I should remove it and use an alias instead [22:08:14] good night [22:08:31] gn8 Platonides [22:16:45] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:16:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:16:55] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:16:55] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:17:14] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:26] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 309152.000000 [22:17:35] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65991 MB (10% inode=99%): [22:17:35] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:17:35] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:17:35] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:17:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 392822 MB (7% inode=63%): [22:17:36] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354508.000000 [22:17:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52491.000000 [22:17:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 255957.000000 [22:17:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 309505.000000 [22:17:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 172950.000000 [22:17:55] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33319.000000 [22:17:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:18:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:18:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:18:05] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:18:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:18:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:06] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:18:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:18:07] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:25] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:18:35] / on thyme is UNKNOWN: NRPE: Unable to read output [22:18:55] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:18:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:18:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 246810.000000 [22:18:55] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [22:19:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:19:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:19:24] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:19:35] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:19:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:19:35] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:31:34] APT on yucca is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:31:55] APT on sage is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:31:56] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:32:05] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:32:25] APT on z-dat-s2-b is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:32:25] APT on z-dat-s5-b is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:38:04] DaBPunkt: tclsh on mayapple seems broken, it points to tclsh8.4 [22:40:56] [[User talk:Florence]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7899&rcid=21745 * Florence * (+241) (i need run my FloBot From Tool Server .) [22:42:11] giftpflanze: fixed [22:42:51] thx :) [22:42:56] np [22:45:08] joins, complains, leaves ;-) [22:47:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:52:24] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 17.413 second response time [23:03:25] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:16:44] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:16:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:17:14] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 310190.000000 [23:17:35] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:17:35] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:17:35] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65664 MB (10% inode=99%): [23:17:35] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:17:35] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 391322 MB (7% inode=63%): [23:17:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354934.000000 [23:17:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43148.000000 [23:17:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 254323.000000 [23:17:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 310528.000000 [23:17:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 172759.000000 [23:17:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:17:55] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26438.000000 [23:18:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:18:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:18:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:18:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:05] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:18:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:18:06] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:35] / on thyme is UNKNOWN: NRPE: Unable to read output [23:18:55] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:18:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:18:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 249010.000000 [23:18:55] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [23:19:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:19:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:19:35] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:19:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:47:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:56:36] @replag