[00:00:12] Load avg. on thyme is OK: OK - load average: 0.96, 0.55, 0.52 [00:00:42] / on thyme is OK: DISK OK - free space: / 9941 MB (49% inode=92%): [00:00:43] /mnt on thyme is OK: DISK OK - free space: / 9941 MB (49% inode=92%): [00:00:43] /sql on thyme is OK: DISK OK - free space: /sql 125026 MB (12% inode=99%): [00:00:44] RAID on thyme is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:02:12] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22913.000000 [00:03:02] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12996.000000 [00:03:03] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:03:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53966.000000 [00:04:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:04:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 234431 MB (4% inode=67%): [00:05:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:05:22] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [00:05:23] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [00:07:02] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 494.000000 [00:07:22] MySQL on thyme is OK: Uptime: 514 Threads: 12 Questions: 16259 Slow queries: 0 Opens: 73 Flush tables: 1 Open tables: 66 Queries per second avg: 31.632 [00:07:22] MySQL slave on thyme is OK: Uptime: 517 Threads: 7 Questions: 18518 Slow queries: 0 Opens: 73 Flush tables: 1 Open tables: 66 Queries per second avg: 35.818 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 494 [00:16:33] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:17:42] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2196.000000 [00:30:10] wikidata replag on z-dat-s2-b is CRITICAL: (Service Check Timed Out) [00:30:11] aliasd on nightshade is CRITICAL: (Service Check Timed Out) [01:05:43] ts have trouble ? unable to log to willow or nightshade and http://toolserver.org/ is broken [01:10:32] I am too, about so send a mail [01:19:03] phe: I just sent a mail to the list [01:19:43] ty Izhidez [01:20:06] Does the tool server has issues? [01:20:16] DeltaQuad Wikipedia * [Toolserver-l] Web services and SSH down? [01:23:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:24:42] Izhidez: you should cc ts-admins probably [01:24:53] or forward now [01:25:26] legoktm: fml it's back up now... [01:25:31] :> [01:25:32] Free Memory on damiana is OK: OK - 84.7% (7097756 kB) free. [01:25:36] sometimes you just need to mail the list [01:26:06] John * Re: [Toolserver-l] Web services and SSH down? [01:28:06] DeltaQuad Wikipedia * Re: [Toolserver-l] Web services and SSH down? [01:28:36] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22817.000000 [01:28:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9669.000000 [01:28:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:29:27] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 235603 MB (4% inode=66%): [01:29:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:29:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:29:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:29:27] PostgreSQL on ptolemy is CRITICAL: (Return code of 137 is out of bounds) [01:29:27] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112046 MB (18% inode=99%): [01:29:36] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 71167 MB (7% inode=99%): [01:29:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:29:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:26] aliasd on nightshade is CRITICAL: Connection refused [01:30:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6557.000000 [01:59:06] MZMcBride * Re: [Toolserver-l] Web services and SSH down? [02:00:25] PostgreSQL on ptolemy is OK: OK - database osm_mapnik (0 sec.) [02:00:44] SMTP on turnera is OK: SMTP OK - 0.032 sec. response time [02:15:11] I'm having trouble logging into willow to renew my account. I can log into nightshade fine, but willow isn't letting me log in to ssh. [02:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:22:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71942.000000 [02:22:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:22:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [02:23:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62344.000000 [02:23:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:29:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:29:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:29:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22318.000000 [02:29:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:29:35] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5279.000000 [02:29:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:29:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:30:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 234224 MB (4% inode=66%): [02:30:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:30:26] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112144 MB (18% inode=99%): [02:31:25] aliasd on nightshade is CRITICAL: Connection refused [02:31:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10217.000000 [02:31:48] Yetanotherx: Logging into nightshade and willow works for me without problems. [03:22:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [03:23:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65944.000000 [03:23:48] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75601.000000 [03:23:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:23:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:29:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:29:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:29:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:29:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8044.000000 [03:29:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4944.000000 [03:30:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:30:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 233885 MB (4% inode=66%): [03:30:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111693 MB (18% inode=99%): [03:30:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:30:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:31:24] aliasd on nightshade is CRITICAL: Connection refused [03:32:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13877.000000 [03:53:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:02:56] Yetanotherx: What error do you get? What command are you runnign? [04:03:15] Susan: try { [04:03:15] XBotDebug.info("AIV", ChatColor.GRAY + "Fixing instructions..."); [04:03:15] String summary = ""; [04:03:15] String content = bot.getParent().getWiki().getPageText(page); [04:03:15] String originalContent = content.toString(); [04:03:16] Calendar time = bot.getParent().getWiki().getTimestamp(); [04:03:17] if (!content.isEmpty()) { [04:03:19] if (matches("===\\s*User-reported\\s*===", content)) { [04:03:21] content = content.replaceAll("", "RE-ADD-HAGERMAN"); // hell if I know... [04:03:24] List reportsToMove = new ArrayList(); [04:03:26] boolean inComment = false; [04:03:28] int reportCount = 0; [04:03:32] String message = ""; [04:03:34] for (String line : content.split("\n")) { [04:03:36] String[] parsed = AIVBot.parseComment(line, inComment); [04:03:38] inComment = Boolean.parseBoolean(parsed[0]); [04:03:40] String remainder = parsed[2]; [04:03:42] String reg = "\\{\\{((?:ip)?vandal|userlinks|user-uaa)\\|\\s*(?!(?:IP ?address|username))"; [04:03:45] if (matches(reg, line, Pattern.CASE_INSENSITIVE)) { [04:03:47] if (inComment) { [04:03:49] // Something other than the example template's in the instructions [04:03:52] // Gonna move it later [04:03:54] reportsToMove.add(line); [04:03:56] reportCount++; [04:03:58] } [04:04:02] } else if (matches(reg, remainder, Pattern.CASE_INSENSITIVE)) { [04:04:04] remainder = remainder.replace("-->", ""); [04:04:06] reportsToMove.add(remainder); [04:04:10] } [04:04:10] } [04:04:12] if (matches("===\\s*User-reported\\s*===\\s+|$)", content, Pattern.DOTALL); [04:04:18] if (m.find()) { [04:04:20] content = m.replaceAll(m.group(1) + bot.getInstructions()); [04:04:22] message = ""; [04:04:24] } [04:04:26] } else { [04:04:28] Matcher m = getMatcher("(===\\s*User-reported\\s*===\\n)", content, Pattern.DOTALL); [04:04:33] if (m.find()) { [04:04:35] content = m.replaceAll(m.group(1) + bot.getInstructions() + "\n"); [04:04:38] message = " Old instructions not found, please check page for problems."; [04:04:41] } [04:04:43] } [04:04:45] String remainingText = AIVBot.getReportSummary(reportCount); [04:04:47] if (!reportsToMove.isEmpty()) { [04:04:49] if (reportsToMove.size() > 50) { [04:04:51] summary = remainingText + " Reset [[WP:AIV/I|instruction block]], WARNING: tried to move more than 50 reports, aborting - check history for lost reports." + message; [04:04:54] } else { [04:04:56] for (String report : reportsToMove) { [04:04:58] if (report.contains("RE-ADD-HAGERMAN")) { [04:05:00] report.replaceAll("RE-ADD-HAGERMAN", ""); [04:05:05] report.replaceAll("~~~~", ""); [04:05:08] } else { [04:05:09] report.replaceAll("~~~~", "~~~~ (Original signature lost - report made inside comment)"); [04:05:12] } [04:05:14] content += report + "\n"; [04:05:16] } [04:05:18] summary = remainingText + " Reset [[WP:AIV/I|instruction block]], " + reportsToMove.size() + " report(s) moved to end of page." + message; [04:05:21] } [04:05:23] } else { [04:05:25] summary = remainingText + " Reset [[WP:AIV/I|instruction block]]." + message; [04:05:28] } [04:05:30] content.replaceAll("RE-ADD-HAGERMAN", ""); [04:05:35] if (!originalContent.equals(bot.getParent().getWiki().getPageText(page))) { [04:05:38] XBotDebug.warn("AIV", ChatColor.BLUE + page + ChatColor.YELLOW + " has changed since we read it, not changing."); [04:05:41] return; [04:05:43] } else { [04:05:45] new Edit(page, content, summary, time).run(bot.getParent(), bot.getRunPage()); [04:05:48] } [04:05:50] XBotDebug.info("AIV", ChatColor.YELLOW + "Reset instruction block on " + page); [04:05:53] } else { [04:05:55] XBotDebug.warn("AIV", "User-reported header not found on " + page); [04:05:57] if (!content.contains("")) { [04:05:59] if (!originalContent.equals(bot.getParent().getWiki().getPageText(page))) { [04:06:04] XBotDebug.warn("AIV", ChatColor.BLUE + page + ChatColor.YELLOW + " has changed since we read it, not changing."); [04:06:06] USE A PASTEBIN [04:06:07] return; [04:06:09] } else { [04:06:11] new Edit(page, content + "\n", "WARNING: User-reported header not found!", time).run(bot.getParent(), bot.getRunPage()); [04:06:14] } [04:06:16] } [04:06:18] } [04:06:19] oh god. [04:06:20] } [04:06:22] } catch (IOException ex) { [04:06:24] XBotDebug.error("AIV", "Could not read from wiki.", ex); [04:06:26] } [04:06:28] Oops [04:06:30] >_< [04:06:34] Susan: Too many authentication failures for soxred93 [04:07:25] Dispenser: It was the wrong this I meant to do [04:07:34] I meant to paste the error from the terminal (Too many authentication failures for soxred93) [04:07:38] But it didn't copy [04:07:47] When I pasted my output, it pasted the last thing from the clipboard. [04:07:53] Which was a large section of the AIVbot code [04:07:55] :-) [04:08:03] At least there weren't any passwords in it. [04:08:24] Yetanotherx: I think acctrenew can only run from Solaris hosts. [04:08:32] Can you ssh from nightshade to willow? [04:08:37] Or does that not work as well? [04:08:41] Lemme try [04:08:46] It sounds like you're going from local to nightshade and from local to willow. [04:08:52] Incidentally, the previous data on the clipboard was a password. :P [04:08:55] Susan: I am. [04:09:26] ssh willow on nightshade works [04:10:19] Susan: Thankfully that code is publicly available on github [04:11:32] Yetanotherx: Once you're into willow, can you run acctrenew? [04:11:36] (Is that the goal?) [04:13:17] Yes, I've renewed. [04:13:27] Not sure why I can't log into willow from my local computer [04:13:45] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3558.000000 [04:18:33] Dunno. It should be an identical ~/.ssh directory. [04:23:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69543.000000 [04:23:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79202.000000 [04:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:23:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:23:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [04:29:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8357.000000 [04:30:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:30:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:30:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 233723 MB (4% inode=66%): [04:30:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111458 MB (18% inode=99%): [04:30:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:30:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:30:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:30:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:31:45] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1702.000000 [04:32:25] aliasd on nightshade is CRITICAL: Connection refused [04:32:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17478.000000 [04:53:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:23:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73145.000000 [05:23:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82802.000000 [05:23:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:23:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [05:30:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:30:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:30:44] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4364.000000 [05:31:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:31:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 233529 MB (4% inode=66%): [05:31:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111282 MB (18% inode=99%): [05:31:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:31:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:32:25] aliasd on nightshade is CRITICAL: Connection refused [05:32:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21080.000000 [05:38:24] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (401412 kB) free! [05:53:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:02:44] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3596.000000 [06:07:42] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1244.000000 [06:23:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76744.000000 [06:23:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [06:24:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86461.000000 [06:24:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:24:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:30:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:30:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:31:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 233202 MB (4% inode=66%): [06:31:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:31:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111077 MB (18% inode=99%): [06:31:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:31:34] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:31:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:31:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:31:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:31:53] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:32:34] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24687.000000 [06:32:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:33:24] aliasd on nightshade is CRITICAL: Connection refused [06:39:23] Free Memory on turnera is CRITICAL: CRITICAL - 2.9% (244732 kB) free! [06:53:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:23:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80344.000000 [07:23:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [07:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:25:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90122.000000 [07:25:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:30:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:30:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:30:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:31:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 232984 MB (4% inode=66%): [07:31:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:31:44] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:31:45] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:31:53] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:31:54] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:32:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110860 MB (18% inode=99%): [07:32:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:32:34] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28288.000000 [07:32:34] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:32:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:33:24] aliasd on nightshade is CRITICAL: Connection refused [07:39:24] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (299260 kB) free! [07:53:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:08:57] [[Special:Log/newusers]] create 10 * Kamalkashyap * (New user account) [08:23:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83945.000000 [08:23:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:25:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93722.000000 [08:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:30:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:30:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:31:44] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:31:54] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:31:54] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:31:54] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:32:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 230776 MB (4% inode=66%): [08:32:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:32:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:33:38] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110664 MB (18% inode=99%): [08:33:38] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31948.000000 [08:33:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:33:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:34:24] aliasd on nightshade is CRITICAL: Connection refused [08:39:33] Free Memory on turnera is CRITICAL: CRITICAL - 2.9% (243224 kB) free! [08:53:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:09:24] aliasd on nightshade is OK: TCP OK - 0.007 second response time on port 984 [500 Not found.] [09:16:24] aliasd on nightshade is CRITICAL: Connection refused [09:20:23] aliasd on nightshade is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [09:23:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87545.000000 [09:23:54] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:25:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 97322.000000 [09:25:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:30:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:30:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:31:45] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:31:54] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:31:54] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:31:54] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:32:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 231050 MB (4% inode=66%): [09:32:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:32:45] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:33:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:33:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:34:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110436 MB (18% inode=99%): [09:34:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35610.000000 [09:40:32] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (176748 kB) free! [09:53:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:06:24] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:07:12] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:07:13] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:46] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:12:53] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:54] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:54] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:59] anyone else having problems ssh'ing in? [10:13:04] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:06] Problems [10:13:12] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:12] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:33] willow at least seems collapsed [10:13:45] aliasd on yarrow is OK: TCP OK - 1.286 second response time on port 984 [500 Not found.] [10:13:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:13:46] /home on hemlock is OK: DISK OK - free space: /home 17309 MB (34% inode=85%): [10:13:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:13:53] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [10:13:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:14:10] i got into willow [10:14:14] cant get into nightshade [10:14:23] got into both [10:14:24] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 42.85, 32.34, 16.10 [10:14:33] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [10:14:46] Apparently getting back to normal [10:14:46] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [10:14:46] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.080 second response time [10:14:46] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.065 second response time [10:16:24] Load avg. on nightshade is WARNING: WARNING - load average: 7.24, 22.31, 14.40 [10:16:33] Load avg. on willow is CRITICAL: CRITICAL - load average: 56.36, 25.95, 11.48 [10:19:24] Load avg. on nightshade is OK: OK - load average: 1.46, 12.74, 12.06 [10:23:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91144.000000 [10:24:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:25:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100921.000000 [10:26:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:30:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:31:45] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:31:53] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:32:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 230512 MB (4% inode=66%): [10:32:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:32:54] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:32:54] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:33:45] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:34:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110124 MB (18% inode=99%): [10:34:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39212.000000 [10:34:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:38:32] Load avg. on willow is WARNING: WARNING - load average: 1.59, 7.84, 19.43 [10:40:32] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (168368 kB) free! [10:41:06] Johannes Kroll * Re: [Toolserver-l] Web services and SSH down? [10:41:06] Marlen Caemmerer * Re: [Toolserver-l] Web services and SSH down? [10:43:32] Load avg. on willow is OK: OK - load average: 2.80, 4.65, 14.74 [10:53:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:01:47] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:08:11] ugh [11:08:12] again [11:10:31] yeps, 404 [11:19:47] sent to the ML [11:19:51] still cant get in [11:20:15] legoktm * Re: [Toolserver-l] Web services and SSH down? [11:44:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:43] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1706.000000 [11:45:00] woo! [11:48:43] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1846.000000 [11:49:03] / on damiana is WARNING: NRPE: Unable to read output [11:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:14] /tmp on damiana is WARNING: NRPE: Unable to read output [11:49:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [11:49:43] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1759.000000 [11:50:03] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3043.000000 [11:50:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:50:04] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3044.000000 [11:50:04] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3048.000000 [11:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:50:13] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 229761 MB (4% inode=66%): [11:50:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host turnera-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [11:50:23] aliasd on nightshade is CRITICAL: Connection refused [11:50:24] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70765 MB (7% inode=99%): [11:50:33] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:50:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:55:11] Johannes Kroll * Re: [Toolserver-l] Web services and SSH down? [11:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000 [11:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [12:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3643.000000 [12:00:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3644.000000 [12:00:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3647.000000 [12:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [12:04:45] I see JIRA still has a short-term memory loss? Need to re-login every 30 seconds [12:05:08] "XSRF Security Token Missing" "Please enable cookies" [12:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109203.000000 [12:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [12:43:53] MySQL on z-dat-s2-b is CRITICAL: Host turnera-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [12:44:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99603.000000 [12:44:07] MySQL slave on z-dat-s2-b is CRITICAL: Host turnera-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [12:44:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109620 MB (18% inode=99%): [12:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:33] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:49:03] / on damiana is WARNING: NRPE: Unable to read output [12:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [12:49:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [12:50:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:50:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:13] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:50:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host turnera-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [12:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 229058 MB (4% inode=66%): [12:50:23] aliasd on nightshade is CRITICAL: Connection refused [12:50:33] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:50:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7227.000000 [12:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7238.000000 [13:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7242.000000 [13:00:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7247.000000 [13:00:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7245.000000 [13:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [13:07:53] MySQL on z-dat-s2-b is OK: Uptime: 90454 Threads: 10 Questions: 86222029 Slow queries: 841 Opens: 711210 Flush tables: 1 Open tables: 256 Queries per second avg: 953.214 [13:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112803.000000 [13:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:44:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103203.000000 [13:44:04] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [13:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109381 MB (18% inode=99%): [13:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:49:03] / on damiana is WARNING: NRPE: Unable to read output [13:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [13:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [13:50:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:50:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50955.000000 [13:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 228535 MB (4% inode=66%): [13:50:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:50:23] aliasd on nightshade is CRITICAL: Connection refused [13:50:33] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:50:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:59:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10827.000000 [13:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10838.000000 [14:00:04] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10843.000000 [14:00:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10847.000000 [14:00:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10844.000000 [14:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [14:13:06] Johannes Kroll * Re: [Toolserver-l] Web services and SSH down? [14:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:43:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:43:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116402.000000 [14:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:44:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106803.000000 [14:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109132 MB (17% inode=99%): [14:44:13] MySQL slave on z-dat-s2-b is CRITICAL: (Service Check Timed Out) [14:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:33] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:49:03] / on damiana is WARNING: NRPE: Unable to read output [14:49:12] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [14:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [14:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:50:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54555.000000 [14:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 228140 MB (4% inode=66%): [14:50:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:50:23] aliasd on nightshade is CRITICAL: Connection refused [14:50:33] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:50:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:59:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14428.000000 [14:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14439.000000 [15:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14443.000000 [15:00:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14447.000000 [15:00:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14444.000000 [15:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [15:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:23:10] Hallo folks! I'm trying to add another ssh-key to my account, without success. I tried that steps: https://wiki.toolserver.org/view/Logging_in#Adding_a_key_from_Unix_or_Linux [15:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120002.000000 [15:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [15:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110403.000000 [15:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108858 MB (17% inode=99%): [15:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:33] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:44:53] MySQL slave on z-dat-s2-b is CRITICAL: (Service Check Timed Out) [15:49:09] / on damiana is WARNING: NRPE: Unable to read output [15:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [15:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [15:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:50:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:50:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 227218 MB (4% inode=66%): [15:50:24] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:50:24] aliasd on nightshade is CRITICAL: Connection refused [15:50:33] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:50:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:50:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58156.000000 [15:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18028.000000 [15:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18039.000000 [16:00:08] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18042.000000 [16:00:08] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18048.000000 [16:00:08] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18044.000000 [16:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [16:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:43:49] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:43:49] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:43:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:43:49] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123603.000000 [16:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [16:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114003.000000 [16:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108555 MB (17% inode=99%): [16:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:33] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:44:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:44:53] MySQL slave on z-dat-s2-b is CRITICAL: (Service Check Timed Out) [16:46:53] MySQL on z-dat-s2-b is CRITICAL: Too many connections [16:47:53] MySQL on z-dat-s2-b is OK: Uptime: 103654 Threads: 501 Questions: 100742572 Slow queries: 937 Opens: 843224 Flush tables: 1 Open tables: 255 Queries per second avg: 971.912 [16:48:09] what's up with s2? [16:49:03] / on damiana is WARNING: NRPE: Unable to read output [16:49:18] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:18] /tmp on damiana is WARNING: NRPE: Unable to read output [16:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:49:44] Load avg. on damiana is WARNING: NRPE: Unable to read output [16:50:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:50:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:23] aliasd on nightshade is CRITICAL: Connection refused [16:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 226625 MB (4% inode=66%): [16:50:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:50:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:50:33] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:51:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Too many connections [16:54:53] MySQL on z-dat-s2-b is CRITICAL: Too many connections [16:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21627.000000 [16:59:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21639.000000 [17:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21642.000000 [17:00:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21647.000000 [17:00:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21644.000000 [17:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [17:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127202.000000 [17:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [17:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117603.000000 [17:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108253 MB (17% inode=99%): [17:44:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:33] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:45:43] MySQL slave on z-dat-s2-b is CRITICAL: Too many connections [17:48:53] MySQL on z-dat-s2-b is CRITICAL: Too many connections [17:49:03] / on damiana is WARNING: NRPE: Unable to read output [17:49:12] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [17:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [17:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:23] aliasd on nightshade is CRITICAL: Connection refused [17:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 226136 MB (4% inode=66%): [17:50:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:50:33] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:51:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Too many connections [17:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25227.000000 [17:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25239.000000 [18:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25242.000000 [18:00:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25248.000000 [18:00:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25244.000000 [18:04:34] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [18:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:27:53] MySQL on z-dat-s2-b is OK: Uptime: 189 Threads: 25 Questions: 1997 Slow queries: 0 Opens: 140 Flush tables: 1 Open tables: 132 Queries per second avg: 10.566 [18:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:43:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:43:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130803.000000 [18:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [18:44:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121203.000000 [18:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107955 MB (17% inode=99%): [18:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27945 [18:49:04] / on damiana is WARNING: NRPE: Unable to read output [18:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [18:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [18:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [18:50:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:50:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:23] aliasd on nightshade is CRITICAL: Connection refused [18:50:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 226323 MB (4% inode=66%): [18:50:24] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:50:44] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:52:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69052.000000 [18:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28828.000000 [18:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28839.000000 [19:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28843.000000 [19:00:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28848.000000 [19:00:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28844.000000 [19:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [19:14:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134403.000000 [19:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [19:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124803.000000 [19:44:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107690 MB (17% inode=99%): [19:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:46:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31280 [19:49:03] / on damiana is WARNING: NRPE: Unable to read output [19:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [19:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:49:44] Load avg. on damiana is WARNING: NRPE: Unable to read output [19:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:29] aliasd on nightshade is CRITICAL: Connection refused [19:50:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 228509 MB (4% inode=66%): [19:50:29] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:50:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72652.000000 [19:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32427.000000 [19:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32438.000000 [20:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32443.000000 [20:00:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32447.000000 [20:00:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32444.000000 [20:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [20:13:42] dab's sending mail but not here... [20:14:05] DaB. * [Toolserver-l] Less time this week and slower sql-s5-user [20:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:21:19] jeremyb_: But you did read the mail? :-) [20:21:42] scfc_de: nope! [20:22:20] scfc_de: now i did [20:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138003.000000 [20:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [20:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 128403.000000 [20:44:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107444 MB (17% inode=99%): [20:44:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29787 [20:49:03] / on damiana is WARNING: NRPE: Unable to read output [20:49:12] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [20:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [20:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:23] aliasd on nightshade is CRITICAL: Connection refused [20:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 227971 MB (4% inode=66%): [20:50:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:50:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76252.000000 [20:59:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36027.000000 [20:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36039.000000 [21:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36042.000000 [21:00:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36048.000000 [21:00:05] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36045.000000 [21:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [21:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:43:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:43:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 141602.000000 [21:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [21:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132003.000000 [21:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107130 MB (17% inode=99%): [21:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28109 [21:49:03] / on damiana is WARNING: NRPE: Unable to read output [21:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [21:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [21:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:23] aliasd on nightshade is CRITICAL: Connection refused [21:50:24] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:50:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 227383 MB (4% inode=66%): [21:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:50:44] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79852.000000 [21:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39627.000000 [21:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39638.000000 [22:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39643.000000 [22:00:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39648.000000 [22:00:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39644.000000 [22:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [22:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:43:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145203.000000 [22:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [22:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135603.000000 [22:44:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106834 MB (17% inode=99%): [22:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:46:06] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29429 [22:49:03] / on damiana is WARNING: NRPE: Unable to read output [22:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [22:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [22:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:50:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:23] aliasd on nightshade is CRITICAL: Connection refused [22:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 230031 MB (4% inode=66%): [22:50:24] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:50:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:51:16] Heh. [22:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83452.000000 [22:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43228.000000 [22:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43238.000000 [23:00:04] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43243.000000 [23:00:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43247.000000 [23:00:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43244.000000 [23:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [23:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 148061.000000 [23:43:54] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [23:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138897.000000 [23:44:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106558 MB (17% inode=99%): [23:44:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28312 [23:49:03] / on damiana is WARNING: NRPE: Unable to read output [23:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:14] /tmp on damiana is WARNING: NRPE: Unable to read output [23:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:49:44] Load avg. on damiana is WARNING: NRPE: Unable to read output [23:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:50:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:23] aliasd on nightshade is CRITICAL: Connection refused [23:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 219113 MB (4% inode=66%): [23:50:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:50:33] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70076 MB (7% inode=99%): [23:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:50:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85994.000000 [23:59:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45427.000000 [23:59:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42454.000000