[00:03:59] Free Memory on turnera is CRITICAL: CRITICAL - 3.5% (294808 kB) free! [00:08:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106909 MB (10% inode=99%): [00:09:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:09:58] Free Memory on turnera is WARNING: WARNING - 5.4% (450408 kB) free! [00:10:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107131 MB (11% inode=99%): [00:33:59] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (416336 kB) free! [00:36:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:46:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35792 MB (5% inode=99%): [00:46:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:48:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:54:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:54:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 486947 MB (9% inode=42%): [00:55:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106455 MB (10% inode=99%): [00:56:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107159 MB (11% inode=99%): [00:56:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:57:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:58:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:59:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:00:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106889 MB (10% inode=99%): [01:09:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:19:58] Free Memory on turnera is WARNING: WARNING - 5.3% (441476 kB) free! [01:21:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (391664 kB) free! [01:29:58] Free Memory on turnera is WARNING: WARNING - 5.2% (434204 kB) free! [01:36:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:46:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:47:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35623 MB (5% inode=99%): [01:48:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:54:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:54:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 489466 MB (9% inode=42%): [01:56:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:58:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:58:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:59:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:59:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (414720 kB) free! [02:00:35] @replag all [02:00:36] Brooke: s1-rr-a: 0s [-0.00 s/s]; s1-user: 0s [-0.00 s/s]; s2-user: 3s [-0.00 s/s]; s2-user-c: 0s [-0.00 s/s]; s3-rr-a: 2m 36s [+0.00 s/s]; s3-user: 2m 36s [+0.00 s/s]; s4-rr-a: 0s [-0.00 s/s]; s4-user: 0s [-0.00 s/s] [02:00:37] Brooke: s5-rr-a: 10s [+0.00 s/s]; s5-user: 10s [+0.00 s/s]; s5-user-c: 0s [-]; s6-rr-a: 4s [-]; s6-user: 4s [-]; s7-rr-a: 1s [+0.00 s/s]; s7-user: 1s [+0.00 s/s] [02:06:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106629 MB (10% inode=99%): [02:10:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:18:58] Free Memory on turnera is WARNING: WARNING - 5.3% (441640 kB) free! [02:36:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106940 MB (11% inode=99%): [02:44:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106792 MB (10% inode=99%): [02:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:46:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:47:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35552 MB (5% inode=99%): [02:48:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:54:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:54:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 487455 MB (9% inode=42%): [02:56:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:59:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:59:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:59:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:03:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.2% (352224 kB) free! [03:10:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:36:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:45:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107130 MB (11% inode=99%): [03:46:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106643 MB (10% inode=99%): [03:46:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:47:58] Free Memory on turnera is WARNING: WARNING - 5.1% (424408 kB) free! [03:48:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35442 MB (5% inode=99%): [03:48:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:54:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 487918 MB (9% inode=42%): [03:55:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:56:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107239 MB (11% inode=99%): [03:56:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:59:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:59:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:59:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:01:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417040 kB) free! [04:04:58] Free Memory on turnera is WARNING: WARNING - 5.4% (455992 kB) free! [04:09:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420264 kB) free! [04:10:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:16:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106051 MB (10% inode=99%): [04:36:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:47:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:48:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35348 MB (5% inode=99%): [04:48:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:49:58] Free Memory on turnera is WARNING: WARNING - 6.1% (508288 kB) free! [04:54:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 487829 MB (9% inode=42%): [04:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:56:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (418568 kB) free! [04:56:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:59:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:59:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:59:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:02:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107499 MB (11% inode=99%): [05:10:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:37:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:44:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:47:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:48:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35221 MB (5% inode=99%): [05:48:20] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:54:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 487715 MB (9% inode=42%): [05:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:57:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.2% (185016 kB) free! [05:57:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:59:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:59:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:59:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:01:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106923 MB (10% inode=99%): [06:04:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107682 MB (11% inode=99%): [06:09:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105389 MB (10% inode=99%): [06:11:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:31:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106002 MB (10% inode=99%): [06:38:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:44:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:47:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:48:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35119 MB (5% inode=99%): [06:49:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:51:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107825 MB (11% inode=99%): [06:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:55:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 487060 MB (9% inode=42%): [06:57:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (178068 kB) free! [06:57:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:00:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:11:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:38:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:47:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34950 MB (5% inode=99%): [07:49:20] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:56:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 486883 MB (9% inode=42%): [07:58:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.2% (271516 kB) free! [07:58:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:00:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:11:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:36:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106923 MB (10% inode=99%): [08:39:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:47:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34759 MB (5% inode=99%): [08:49:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:54:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107361 MB (11% inode=99%): [08:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:57:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 486680 MB (9% inode=42%): [08:58:59] Free Memory on turnera is CRITICAL: CRITICAL - 3.4% (287216 kB) free! [08:58:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:00:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:00:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:00:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 76686 MB (7% inode=98%): [09:11:09] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 76685 MB (7% inode=98%): [09:11:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:11:38] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38171 MB (9% inode=99%): [09:15:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106872 MB (10% inode=99%): [09:16:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107052 MB (11% inode=99%): [09:39:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:41:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106921 MB (10% inode=99%): [09:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:48:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:49:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34537 MB (5% inode=99%): [09:49:20] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:57:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 482791 MB (9% inode=42%): [09:58:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.2% (264416 kB) free! [09:58:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:00:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:00:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:00:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:11:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:14:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106982 MB (11% inode=99%): [10:27:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106918 MB (10% inode=99%): [10:39:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:46:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2141.000000 [10:48:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34339 MB (5% inode=99%): [10:49:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:57:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 456597 MB (8% inode=40%): [10:59:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (238632 kB) free! [10:59:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:00:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107191 MB (11% inode=99%): [11:00:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:08:38] hello all [11:11:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:19:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [11:27:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106395 MB (10% inode=99%): [11:31:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108168 MB (11% inode=99%): [11:32:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106915 MB (10% inode=99%): [11:40:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:48:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34157 MB (5% inode=99%): [11:49:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:54:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106884 MB (10% inode=99%): [11:57:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 477466 MB (9% inode=42%): [11:58:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:59:57] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (255140 kB) free! [12:00:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:00:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:00:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:07:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107325 MB (11% inode=99%): [12:08:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106890 MB (10% inode=99%): [12:11:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:20:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6621.000000 [12:41:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:48:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:49:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33908 MB (5% inode=99%): [12:50:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:57:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 477284 MB (9% inode=42%): [12:58:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:59:57] Free Memory on turnera is CRITICAL: CRITICAL - 3.5% (296624 kB) free! [12:59:58] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2183 [13:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:00:19] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2193.000000 [13:00:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:00:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:11:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:20:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6224.000000 [13:24:58] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3640 [13:25:19] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3650.000000 [13:41:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:45:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:48:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33652 MB (5% inode=99%): [13:50:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:57:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 477053 MB (9% inode=42%): [13:58:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:59:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.2% (268552 kB) free! [14:00:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:00:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:00:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:00:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:11:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:20:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3882.000000 [14:24:57] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7007 [14:25:19] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7017.000000 [14:28:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3546.000000 [14:36:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107501 MB (11% inode=99%): [14:41:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:48:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:49:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33428 MB (5% inode=99%): [14:50:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:58:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:58:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 476827 MB (9% inode=42%): [14:59:59] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (309584 kB) free! [15:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:00:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:00:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:10:58] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1776.000000 [15:11:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:16:58] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3482 [15:17:08] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3374.000000 [15:19:58] MySQL slave on daphne is OK: Uptime: 18451485 Threads: 16 Questions: 1465397537 Slow queries: 633204 Opens: 83516 Flush tables: 3 Open tables: 1085 Queries per second avg: 79.418 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1267 [15:20:08] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1118.000000 [15:23:24] @replag [15:23:25] liangent: s1-rr-a: 12s [+0.00 s/s]; s1-user: 13s [+0.00 s/s]; s2-user: 3h 45m 8s [+0.28 s/s]; s2-user-c: 19m 14s [+0.02 s/s]; s3-rr-a: 53s [-0.00 s/s]; s3-user: 53s [-0.00 s/s]; s5-user-c: 19m 12s [+0.02 s/s]; s6-rr-a: 10s [+0.00 s/s] [15:23:26] liangent: s6-user: 10s [+0.00 s/s] [15:41:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:47:24] this is silly [15:47:48] why could a simple select server from wiki where dbname='commonswiki'; be blocking? [15:48:26] it's not like someone would be writing there [15:48:38] * Platonides kills after 8 minutes [15:48:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33215 MB (5% inode=99%): [15:50:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:58:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:58:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 476521 MB (9% inode=42%): [16:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:00:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:00:59] Free Memory on turnera is CRITICAL: CRITICAL - 2.7% (227268 kB) free! [16:00:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:03:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106915 MB (10% inode=99%): [16:11:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:24:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:41:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:48:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32996 MB (5% inode=99%): [16:49:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107108 MB (11% inode=99%): [16:50:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:53:08] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198855 MB (20% inode=99%): [16:54:47] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:58:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:58:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 476320 MB (8% inode=42%): [17:00:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:00:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:00:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:00:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.0% (331848 kB) free! [17:00:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:04:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106924 MB (10% inode=99%): [17:05:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107078 MB (11% inode=99%): [17:11:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:12:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106917 MB (10% inode=99%): [17:21:48] @replag [17:21:50] giftpflanze: s1-rr-a: 29s [+0.00 s/s]; s1-user: 31s [+0.00 s/s]; s2-user: 4h 45m 9s [+0.51 s/s]; s6-rr-a: 15s [+0.00 s/s]; s6-user: 15s [+0.00 s/s] [17:22:00] hm [17:23:08] /sql on thyme is OK: DISK OK - free space: /sql 207561 MB (21% inode=99%): [17:41:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:48:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:49:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32759 MB (5% inode=99%): [17:50:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:58:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:58:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 476036 MB (8% inode=41%): [18:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:01:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:01:59] Free Memory on turnera is CRITICAL: CRITICAL - 2.5% (213636 kB) free! [18:01:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:06:45] @replag [18:06:46] jimmyxu: s2-user: 5h 4m 10s [+0.42 s/s]; s3-rr-a: 15s [-0.00 s/s]; s3-user: 15s [-0.00 s/s] [18:12:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:29:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106175 MB (10% inode=99%): [18:31:26] Erik Moeller * Re: [Toolserver-l] Future of the toolserver [18:42:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32462 MB (5% inode=99%): [18:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:50:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:58:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:58:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 472811 MB (8% inode=41%): [19:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:01:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:01:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (247744 kB) free! [19:01:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:10:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2126.000000 [19:12:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:39:53] DaBPunkt, ssh to ptolemy still gives me a connection closed error [19:41:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [19:42:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:45:16] dschwen: who told you that you can login on ptolemy? [19:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:45:51] it was the main request in the ticket [19:46:03] sorry if that was not clear [19:46:19] but it was the idea, and Kai mentioned it in his comment as well [19:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32221 MB (5% inode=99%): [19:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:50:02] dschwen: ok, wait a moment [19:50:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:51:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3592.000000 [19:54:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [19:56:09] dschwen: ok, should work within 1h [19:58:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 476067 MB (8% inode=41%): [19:59:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:00:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:01:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:01:59] Free Memory on turnera is CRITICAL: CRITICAL - 2.6% (220800 kB) free! [20:01:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:06:45] thx [20:09:28] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1981.000000 [20:09:59] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1973 [20:13:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:14:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2659.000000 [20:20:15] the web servers seem down [20:20:54] let me see [20:23:18] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:18] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [20:23:48] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:24:09] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [20:29:58] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1773.000000 [20:31:58] MySQL slave on rosemary is OK: Uptime: 14966074 Threads: 9 Questions: 6659514348 Slow queries: 2066536 Opens: 288873 Flush tables: 6 Open tables: 3869 Queries per second avg: 444.974 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [20:32:29] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1782.000000 [20:36:09] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/4:UP:1 UP: OK [20:36:19] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/2:UP:1 UP: OK [20:40:30] ETA for when toolserver is back up? [20:41:28] DaBPunkt said he was working on it a while back [20:41:33] Riley: You might want to say toolserver webservers instead of "Toolserver" ;-) [20:42:00] Whatever ;) [20:42:05] Thanks. [20:42:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:42:37] Load avg. on hemlock is CRITICAL: Connection refused by host [20:42:59] /aux0 on hemlock is CRITICAL: Connection refused by host [20:42:59] SSH on hemlock is CRITICAL: Connection refused [20:43:09] CAM on hemlock is CRITICAL: Connection refused by host [20:43:09] FMA on hemlock is CRITICAL: ERROR - unexpected output from snmpwalk [20:43:19] SMTP on hemlock is CRITICAL: Connection refused [20:43:19] /home on hemlock is CRITICAL: Connection refused by host [20:43:19] Environment IPMI on hemlock is CRITICAL: Connection refused by host [20:43:19] /tmp on hemlock is CRITICAL: Connection refused by host [20:43:19] / on hemlock is CRITICAL: Connection refused by host [20:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31994 MB (5% inode=99%): [20:49:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:50:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:50:34] bye bye hemlock.... [20:56:31] hemlock's disc-array is making problems again. We try to fix it, but it will take some time [20:58:06] thank you DaBPunkt! [20:59:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:00:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:01:03] DaBPunkt: Is that the disk array with the cable/connector from Mars? [21:01:31] (for console) [21:02:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:02:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.5% (296952 kB) free! [21:02:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:09:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107321 MB (11% inode=99%): [21:11:09] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 75150 MB (7% inode=98%): [21:11:38] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39178 MB (9% inode=99%): [21:11:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 75140 MB (7% inode=98%): [21:13:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:18:58] Free Memory on turnera is WARNING: WARNING - 5.1% (423272 kB) free! [21:23:19] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:47] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:15] DaBPunkt, does not work yet :-( [21:42:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:42:38] Load avg. on hemlock is CRITICAL: Connection refused by host [21:42:58] /aux0 on hemlock is CRITICAL: Connection refused by host [21:42:58] SSH on hemlock is CRITICAL: Connection refused [21:43:08] CAM on hemlock is CRITICAL: Connection refused by host [21:43:08] FMA on hemlock is CRITICAL: ERROR - unexpected output from snmpwalk [21:43:18] SMTP on hemlock is CRITICAL: Connection refused [21:43:18] /home on hemlock is CRITICAL: Connection refused by host [21:43:18] Environment IPMI on hemlock is CRITICAL: Connection refused by host [21:43:18] /tmp on hemlock is CRITICAL: Connection refused by host [21:43:18] / on hemlock is CRITICAL: Connection refused by host [21:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31718 MB (5% inode=99%): [21:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:50:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:59:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:00:57] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (234464 kB) free! [22:01:09] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.009 second response time [22:01:38] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.016 second response time [22:01:39] 2h have passed now. Is this still going to happen, or did something go wrong? [22:02:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:02:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:03:50] where did /mnt/user-store go? [22:04:20] needs fsck which takes quite long [22:04:27] digging in backlog [22:05:06] we need new machines [22:12:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106898 MB (10% inode=99%): [22:13:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107661 MB (11% inode=99%): [22:14:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:23:27] DaB. * [Toolserver-announce] /mnt/user-store is away [22:23:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106828 MB (10% inode=99%): [22:24:56] o_O [22:24:58] Free Memory on turnera is WARNING: WARNING - 5.2% (433980 kB) free! [22:25:11] DaBPunkt: but no data will be lost, right? [22:26:46] I guess not [22:27:11] no ptolemy for me [22:27:52] all this should be logged and forwarded to relevant people to illustrate the need for new hw [22:33:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (411732 kB) free! [22:34:57] Free Memory on turnera is WARNING: WARNING - 5.8% (483996 kB) free! [22:37:00] dschwen: there will be no puppet until hemlock is fully recovered. So no osm-titlkes and no puppet. So no login for you [22:37:23] ok, I see , thanks [22:42:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105278 MB (10% inode=99%): [22:45:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:49:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31568 MB (5% inode=99%): [22:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:50:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:54:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.1% (259172 kB) free! [22:59:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:02:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:02:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:04:58] Free Memory on turnera is WARNING: WARNING - 5.2% (437464 kB) free! [23:08:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (378108 kB) free! [23:15:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:17:39] nacht ts [23:25:58] Free Memory on turnera is WARNING: WARNING - 5.2% (436712 kB) free! [23:33:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (402872 kB) free! [23:34:59] Free Memory on turnera is WARNING: WARNING - 5.5% (463736 kB) free! [23:34:59] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2032 [23:35:28] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2048.000000 [23:42:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31403 MB (5% inode=99%): [23:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:51:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:59:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output