[00:00:28] oh oh [00:00:31] einen moment [00:00:33] and now? [00:01:38] oooch, nix [00:01:45] einfach abwarten [00:01:58] wird bestimmt bald wieder alles gut. Huestel. [00:02:40] mm, it works perfect for me [00:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:06:25] Betacommand: works it for you? [00:07:19] DaBPunkt: been wrapped up in an issue, If I continue to have issues later Ill let you kniw [00:07:44] it will work for him [00:07:44] I will leave for bed soon [00:07:53] I fucked up [00:07:55] Free Memory on damiana is OK: OK - 7.5% (627472 kB) free. [00:08:33] I applied a patch to the wikiminiatlas client code that was supposed to check if WIWOSM data exists for an article [00:09:00] to display the globe icon in articles that have no coordinate templates but WIWOSM overlay data [00:09:08] (such as train lines) [00:09:54] unfortunately, as it appears, the flood of WIWOSM requests blasted the nginx servers to shreds [00:10:04] sorry! [00:10:24] well, in my defense, this was resolved fairly quickly... [00:10:36] ...please don't yell at me DaB.... [00:11:12] * dschwen is curled up crying on the floorx [00:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:13:35] * valhallasw gives dschwen a cookie [00:14:11] * dschwen doesn't deserve a cookie :-( [00:14:31] the only way to never fuck up is to never do anything [00:14:37] I guess "with great power comes great responsibility". [00:16:45] I wonder why DaBPunkt is not commenting anymore... ...he is probably busy deleting my account :-( [00:17:12] Or he face-palmed himself unconscious. [00:27:41] dschwen: [00:07] I will leave for bed soon [00:27:47] he must be already dreaming :P [00:28:32] I get to live another day [00:28:55] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (379316 kB) free! [00:29:05] dreaming of users who let me play HalfLife… ;) [00:29:51] good news, he is alive! we where thinking in sending an ambulance [00:29:56] Free Memory on damiana is WARNING: WARNING - 5.7% (477060 kB) free! [00:29:56] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 242 bytes in 0.658 second response time [00:36:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1208604.000000 [00:38:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87288 MB (14% inode=99%): [00:38:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 118934 MB (2% inode=52%): [00:38:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43900 [00:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:39:08] night ts! [00:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43789.000000 [00:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:45:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.34, 20.91, 20.91 [00:48:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:48:55] Free Memory on damiana is OK: OK - 7.1% (590792 kB) free. [01:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:23:25] * Dispenser wonders about Half-Life 3 [01:28:45] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 72535 MB (7% inode=99%): [01:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 242 bytes in 0.558 second response time [01:36:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1196641.000000 [01:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86460 MB (14% inode=99%): [01:38:45] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 118094 MB (2% inode=52%): [01:38:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41173 [01:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41142.000000 [01:40:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:45:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.75, 21.60, 21.63 [01:48:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:50:44] Load avg. on ortelius is WARNING: WARNING - load average: 26.07, 20.26, 15.36 [02:00:45] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 31.89, 23.30, 18.34 [02:01:44] Load avg. on ortelius is WARNING: WARNING - load average: 26.68, 23.20, 18.62 [02:03:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:24:45] Load avg. on ortelius is OK: OK - load average: 4.38, 10.90, 14.81 [02:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.518 second response time [02:31:58] nacht ts [02:36:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1178933.000000 [02:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86948 MB (14% inode=99%): [02:38:45] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 116639 MB (2% inode=52%): [02:38:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39098 [02:38:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39061.000000 [02:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:41:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:45] Load avg. on ortelius is WARNING: WARNING - load average: 16.40, 17.54, 15.88 [02:45:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.19, 22.14, 22.14 [02:48:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:52:45] Load avg. on ortelius is OK: OK - load average: 5.41, 12.68, 14.69 [03:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:12:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:29:55] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.512 second response time [03:36:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1167225.000000 [03:38:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86840 MB (14% inode=99%): [03:38:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36202 [03:38:45] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 115740 MB (2% inode=52%): [03:38:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:39:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36218.000000 [03:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:41:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:45:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.55, 23.39, 23.25 [03:48:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:04:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:04:40] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [04:04:40] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [04:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:18:42] MySQL on z-dat-s3-a is OK: Uptime: 1208 Threads: 4 Questions: 142 Slow queries: 1 Opens: 66 Flush tables: 1 Open tables: 56 Queries per second avg: 0.117 [04:18:42] MySQL slave on z-dat-s3-a is OK: Uptime: 1208 Threads: 4 Questions: 143 Slow queries: 1 Opens: 66 Flush tables: 1 Open tables: 56 Queries per second avg: 0.118 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1212 [04:30:04] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.515 second response time [04:36:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:37:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:38:01] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1157624.000000 [04:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 115315 MB (2% inode=52%): [04:38:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35879 [04:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:39:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86803 MB (14% inode=99%): [04:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35807.000000 [04:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 43.41, 29.96, 26.77 [04:49:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:04:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:12:24] JIRA is still down :P [05:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:31:01] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.584 second response time [05:36:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:37:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1153510.000000 [05:38:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36342 [05:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 115147 MB (2% inode=52%): [05:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86705 MB (14% inode=99%): [05:40:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36353.000000 [05:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.30, 23.62, 23.57 [05:49:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:04:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:08:00] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (365240 kB) free! [06:13:00] Free Memory on damiana is WARNING: WARNING - 5.2% (436848 kB) free! [06:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:17:01] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (240304 kB) free! [06:32:07] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.530 second response time [06:36:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:37:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1147696.000000 [06:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 114842 MB (2% inode=52%): [06:38:51] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33239 [06:38:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86596 MB (14% inode=99%): [06:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33123.000000 [06:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.38, 22.79, 23.30 [06:49:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:50:36] [[Special:Log/newusers]] create 10 * روبہ خوش آمدید * (New user account) [07:04:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:32:00] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.511 second response time [07:33:54] [[Special:Log/newusers]] create 10 * Houseplaque * (New user account) [07:34:59] [[User talk:Houseplaque]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7795&rcid=21500 * Houseplaque * (+1478) (Created page with "Adding a personalised house plaque to the house really helps it be prominent and generates a great first impression. But the large number of homeowners ready to forget about maki...") [07:36:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:37:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1135527.000000 [07:38:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31348 [07:38:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 114510 MB (2% inode=52%): [07:38:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86485 MB (14% inode=99%): [07:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31278.000000 [07:40:07] [[User talk:Houseplaque]] ! 10https://wiki.toolserver.org/w/index.php?diff=7796&oldid=7795&rcid=21501 * Legoktm * (-1478) (spam) [07:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.23, 24.13, 23.78 [07:49:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:04:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:32:00] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.517 second response time [08:36:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:37:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1121864.000000 [08:38:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29689 [08:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 114160 MB (2% inode=52%): [08:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86344 MB (14% inode=99%): [08:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29674.000000 [08:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:45:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.24, 24.64, 24.71 [08:49:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:04:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:32:00] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.663 second response time [09:36:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1116822.000000 [09:38:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:38:49] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27445 [09:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 113764 MB (2% inode=52%): [09:38:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86143 MB (14% inode=99%): [09:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27376.000000 [09:40:59] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Houseplaque10]]": spam) [09:41:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:41:15] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Houseplaque10]] with an expiry time of infinite (account creation disabled): spam) [09:41:40] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:WhiteyTrece85010]]": spam) [09:41:54] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:WhiteyTrece85010]] with an expiry time of infinite (account creation disabled): spam) [09:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:43:50] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:2squareTMG10]]": not needed) [09:44:06] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Autoshop10]]": not needed) [09:44:13] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Bernardinebeckwith1910]]": not needed) [09:44:24] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Certiifcatemare10]]": not needed) [09:44:32] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Digitalserver10]]": not needed) [09:44:40] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Dogsantamonica10]]": not needed) [09:44:48] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Dogwashington10]]": not needed) [09:44:55] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Filamatry3510]]": not needed) [09:45:15] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Greenvillerealestatesc10]]": not needed) [09:45:21] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Ingeborgbashir0710]]": not needed) [09:45:26] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Nancyng10]]": not needed) [09:45:33] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Onlinecasino10]]": not needed) [09:45:39] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Paddleboarding10]]": not needed) [09:45:44] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Potvintricia9910]]": not needed) [09:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.25, 27.03, 25.49 [09:46:02] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Rebecatatro10]]": not needed) [09:46:07] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Supsurf10]]": not needed) [09:46:13] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Supsurfing10]]": not needed) [09:46:18] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Supsurfs10]]": not needed) [09:46:26] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Surfboard10]]": not needed) [09:46:41] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Surfboards10]]": not needed) [09:46:51] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Washwaresm10]]": not needed) [09:46:55] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Weddingg10]]": not needed) [09:47:46] heh [09:49:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:49:12] [[Template:Getting started navbox]] 10https://wiki.toolserver.org/w/index.php?diff=7797&oldid=7734&rcid=21528 * MZMcBride * (+464) (rv) [09:49:48] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User:Betacommand/Edit count10]]": not needed) [09:53:00] /sql on cassia is WARNING: DISK WARNING - free space: /sql 113907 MB (9% inode=99%): [10:04:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:32:00] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.776 second response time [10:37:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1112779.000000 [10:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:38:51] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24589 [10:38:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 113308 MB (2% inode=52%): [10:38:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85884 MB (14% inode=99%): [10:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24467.000000 [10:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:45:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.25, 22.77, 23.64 [10:49:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:32:01] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.661 second response time [11:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1111003.000000 [11:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:38:51] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21913 [11:38:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 112885 MB (2% inode=52%): [11:38:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85631 MB (14% inode=99%): [11:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21924.000000 [11:41:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.80, 23.06, 22.93 [11:49:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:59:40] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2164 [12:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:05:00] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2147 [12:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:11:40] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [12:12:00] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [12:13:01] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [12:13:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:13:20] RAID on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:14:29] wikidata replag on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [12:16:21] Environment IPMI on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:40] / on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:40] /tmp on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:40] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:40] / on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:50] /tmp on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:50] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:50] / on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:50] Load avg. on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:50] /tmp on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:50] / on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:50] /sql on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:51] /tmp on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:51] / on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:52] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:52] Load avg. on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:16:53] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:00] Load avg. on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:17:00] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:00] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:00] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:00] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:01] Load avg. on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:17:01] Load avg. on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:17:02] /tmp on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:17:10] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:10] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:40] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [12:17:50] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:00] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:09] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:50] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [12:18:50] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [12:19:00] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [12:19:00] wikidata replag on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [12:19:30] uhohs... [12:19:59] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [12:20:01] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:50] NTP on hyacinth is OK: NTP OK: Offset 7.3e-05 secs [12:21:20] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:21:21] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:21:40] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:21:40] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:00] Load avg. on hyacinth is OK: OK - load average: 0.00, 0.05, 0.34 [12:22:10] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:10] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:10] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:10] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:10] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:10] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:11] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:11] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:12] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:12] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:13] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:14] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:30] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:30] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:30] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:30] MySQL on z-dat-s4-a is OK: Uptime: 2121009 Threads: 2 Questions: 260483997 Slow queries: 77 Opens: 391 Flush tables: 7 Open tables: 119 Queries per second avg: 122.811 [12:22:40] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2264 [12:22:40] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:22:40] SMTP on z-dat-s3-a is OK: SMTP OK - 0.003 sec. response time [12:22:40] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:22:40] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 33381 MB (99% inode=98%): [12:22:41] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 294403 MB (36% inode=99%): [12:22:41] / on z-dat-s7-a is OK: DISK OK - free space: / 6864 MB (22% inode=84%): [12:22:42] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [12:22:42] / on z-dat-s3-a is OK: DISK OK - free space: / 6864 MB (22% inode=84%): [12:22:43] Load avg. on z-dat-s6-a is OK: OK - load average: 0.27, 0.10, 0.34 [12:22:43] /tmp on hyacinth is OK: DISK OK - free space: /tmp 37415 MB (99% inode=98%): [12:22:44] / on z-dat-s4-a is OK: DISK OK - free space: / 6864 MB (22% inode=84%): [12:23:00] SMTP on z-dat-s7-a is OK: SMTP OK - 0.002 sec. response time [12:23:01] SMTP on z-dat-s4-a is OK: SMTP OK - 0.012 sec. response time [12:23:01] SMTP on z-dat-s6-a is OK: SMTP OK - 0.046 sec. response time [12:23:01] Load avg. on z-dat-s3-a is OK: OK - load average: 0.74, 0.21, 0.38 [12:23:01] Load avg. on z-dat-s4-a is OK: OK - load average: 0.77, 0.22, 0.38 [12:23:01] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 47607 MB (99% inode=98%): [12:29:40] MySQL slave on z-dat-s4-a is OK: Uptime: 2121431 Threads: 1 Questions: 260523011 Slow queries: 77 Opens: 391 Flush tables: 7 Open tables: 119 Queries per second avg: 122.805 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1751 [12:32:00] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.585 second response time [12:32:30] MySQL on z-dat-s7-a is OK: Uptime: 582 Threads: 1 Questions: 24 Slow queries: 0 Opens: 32 Flush tables: 1 Open tables: 22 Queries per second avg: 0.41 [12:32:50] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2715.000000 [12:33:20] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3438 [12:37:29] MySQL on z-dat-s6-a is OK: Uptime: 882 Threads: 6 Questions: 1646 Slow queries: 1 Opens: 108 Flush tables: 1 Open tables: 93 Queries per second avg: 1.866 [12:37:40] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2732.000000 [12:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1109800.000000 [12:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:38:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15818 [12:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 112370 MB (2% inode=52%): [12:38:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:39:20] MySQL slave on z-dat-s7-a is OK: Uptime: 993 Threads: 5 Questions: 133351 Slow queries: 8 Opens: 547 Flush tables: 1 Open tables: 520 Queries per second avg: 134.291 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1580 [12:39:41] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3451 [12:39:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85382 MB (14% inode=99%): [12:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15661.000000 [12:41:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:43:40] MySQL on z-dat-s3-a is OK: Uptime: 1251 Threads: 8 Questions: 394 Slow queries: 0 Opens: 134 Flush tables: 1 Open tables: 126 Queries per second avg: 0.314 [12:44:40] MySQL slave on z-dat-s6-a is OK: Uptime: 1312 Threads: 8 Questions: 616467 Slow queries: 22 Opens: 221 Flush tables: 1 Open tables: 202 Queries per second avg: 469.868 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1493 [12:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.12, 26.74, 26.20 [12:47:50] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [12:49:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:52:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3632.000000 [13:04:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:10:41] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3589 [13:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:19:40] MySQL slave on z-dat-s3-a is OK: Uptime: 3411 Threads: 16 Questions: 1644724 Slow queries: 43 Opens: 20125 Flush tables: 1 Open tables: 12411 Queries per second avg: 482.182 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1779 [13:29:40] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 73652 MB (7% inode=99%): [13:32:00] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.505 second response time [13:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1110580.000000 [13:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:38:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10246 [13:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 111450 MB (2% inode=51%): [13:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85092 MB (13% inode=99%): [13:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10050.000000 [13:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:45:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.57, 27.17, 27.17 [13:47:50] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7215.000000 [13:49:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:52:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7232.000000 [14:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:32:01] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.510 second response time [14:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1109427.000000 [14:38:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:38:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4288 [14:38:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 110472 MB (2% inode=51%): [14:38:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [14:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84803 MB (13% inode=99%): [14:40:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4182.000000 [14:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.87, 29.44, 28.74 [14:47:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10815.000000 [14:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:51:50] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3527 [14:52:01] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3493.000000 [14:52:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10831.000000 [15:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:10:50] MySQL slave on rosemary is OK: Uptime: 4901377 Threads: 15 Questions: 3296401607 Slow queries: 1144052 Opens: 312764 Flush tables: 1 Open tables: 3824 Queries per second avg: 672.546 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1711 [15:11:00] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1680.000000 [15:13:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:32:10] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.639 second response time [15:34:25] [[Special:Log/newusers]] create 10 * IdeasBot * (New user account) [15:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1106847.000000 [15:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 109876 MB (2% inode=51%): [15:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84497 MB (13% inode=99%): [15:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:42:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.02, 28.52, 28.44 [15:47:50] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14416.000000 [15:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:52:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14432.000000 [16:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:13:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:24:44] [[User:Dab/s2-userdatabaes]] 10https://wiki.toolserver.org/w/index.php?diff=7798&oldid=7764&rcid=21531 * Dab * (+77) (closed) [16:32:10] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 244 bytes in 0.683 second response time [16:33:26] [[Special:Log/move]] move 10 * Dab * (moved [[02User:Dab/s2-userdatabaes10]] to [[User:Dab/s2-userdatabases]]: typo) [16:38:01] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1104688.000000 [16:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 108730 MB (2% inode=51%): [16:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84199 MB (13% inode=99%): [16:41:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.76, 29.04, 28.82 [16:47:50] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18015.000000 [16:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:52:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18032.000000 [17:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:13:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:32:10] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 245 bytes in 0.512 second response time [17:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1099647.000000 [17:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 107997 MB (2% inode=51%): [17:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83901 MB (13% inode=99%): [17:41:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:42:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.66, 28.02, 28.22 [17:46:47] [[Special:Log/newusers]] create 10 * Betty Waltun * (New user account) [17:47:05] DaB. * Re: [Toolserver-announce] [Toolserver-l] s2/s5 will be read-only at Saturday (was: Re: Split of s2/s5 (cassia)) [17:47:50] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21616.000000 [17:49:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:52:39] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21632.000000 [17:56:03] [[User talk:Betty Waltun]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7801&rcid=21534 * Betty Waltun * (+877) (Created page with "[http://www.finanziellen-freiraum.de/ratenkredit.html Zinsgünstige Kreditangebote] [http://www.finanziellen-freiraum.de/versicherungsvergleich.html sehr gefragt im Onlinebereich...") [17:56:57] [[Special:Log/move]] move 10 * Betty Waltun * (moved [[02User talk:Betty Waltun10]] to [[Nur mit einer ausreichende Bonität bekommt man einen Online Kredit]]) [17:57:30] [[Nur mit einer ausreichende Bonität bekommt man einen Online Kredit]] ! 10https://wiki.toolserver.org/w/index.php?diff=7804&oldid=7802&rcid=21536 * Betty Waltun * (+5) () [18:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:13:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:32:10] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 245 bytes in 0.534 second response time [18:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1092063.000000 [18:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 107421 MB (2% inode=51%): [18:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83583 MB (13% inode=99%): [18:41:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:42:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.44, 28.71, 29.04 [18:47:50] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22087.000000 [18:49:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:52:39] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20817.000000 [19:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:13:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:25:40] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3234.000000 [19:27:40] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1653.000000 [19:27:50] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3460.000000 [19:30:54] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1451.000000 [19:32:10] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 245 bytes in 0.529 second response time [19:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1085617.000000 [19:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 106877 MB (2% inode=51%): [19:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83326 MB (13% inode=99%): [19:41:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:43:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:45:05] Platonides * Re: [Toolserver-l] s2/s5 will be read-only at Saturday [19:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.88, 28.62, 28.77 [19:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:13:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:32:10] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 245 bytes in 0.510 second response time [20:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1080180.000000 [20:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 106355 MB (1% inode=51%): [20:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:39:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82986 MB (13% inode=99%): [20:41:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:43:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.29, 29.45, 28.97 [20:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:04:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:13:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:32:10] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 245 bytes in 0.515 second response time [21:38:01] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1075688.000000 [21:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 105880 MB (1% inode=51%): [21:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:39:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82723 MB (13% inode=99%): [21:41:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:43:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.20, 30.09, 30.31 [21:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:53:00] /sql on cassia is WARNING: DISK WARNING - free space: /sql 113444 MB (9% inode=99%): [22:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:13:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:32:10] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 245 bytes in 0.520 second response time [22:35:33] @replag [22:35:33] Merlissimo: s1-rr-a: 13s [-0.00 s/s]; s1-rr-a-wd: 41s [-0.00 s/s]; s1-user: 3m 24s [-0.58 s/s]; s1-user-wd: 45s [-0.00 s/s]; s2-user-c: error; s2-user-wd: 42s [-0.00 s/s]; s3-user: 11s [-0.00 s/s]; s4-user-wd: 43s [-0.00 s/s] [22:35:35] Merlissimo: s5-user-c: 1w 5d 9h 32m 47s [-1.81 s/s]; s5-user-wd: 39s [-0.00 s/s]; s6-user-wd: 39s [-0.00 s/s]; s7-user-wd: 39s [-0.00 s/s] [22:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1070975.000000 [22:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 105537 MB (1% inode=51%): [22:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82489 MB (13% inode=99%): [22:41:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:43:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.90, 29.83, 30.02 [22:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:04:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:04:10] jira.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Proxy Error - 150 bytes in 0.517 second response time [23:05:09] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 245 bytes in 0.522 second response time [23:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:13:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:38:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1066551.000000 [23:38:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:38:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 103809 MB (1% inode=51%): [23:38:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:39:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82274 MB (13% inode=99%): [23:41:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:43:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:45:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.17, 30.11, 29.88 [23:49:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output