[00:06:44] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3613 [00:07:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3642.000000 [00:11:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:33:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12166.000000 [00:35:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77801 MB (12% inode=99%): [00:36:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:04] aliasd on mayapple is CRITICAL: Connection refused [00:36:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:36:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:36:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 160049 MB (3% inode=68%): [00:36:34] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:36:34] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:36:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:37:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:37:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:39:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:51:24] Free Memory on damiana is WARNING: WARNING - 6.7% (560044 kB) free! [00:56:24] Free Memory on damiana is OK: OK - 7.3% (614968 kB) free. [01:06:44] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6264 [01:07:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6279.000000 [01:08:24] Free Memory on damiana is WARNING: WARNING - 5.6% (470820 kB) free! [01:08:32] @replag [01:08:34] DaBPunkt: s1-rr-a-wd: 25s [-]; s1-user-wd: 22s [-]; s2-rr: 1m 32s [+0.01 s/s]; s2-user: 1m 32s [+0.01 s/s]; s2-user-c: error; s2-user-wd: 3h 15m 44s [+0.08 s/s]; s3-user: 1m 7s [+0.01 s/s]; s3-user-wd: 17s [-0.00 s/s] [01:08:35] DaBPunkt: s4-user-wd: 20m 26s [+0.55 s/s]; s5-user: 1h 45m 26s [+0.59 s/s]; s5-user-c: error; s6-user-wd: 25s [+0.00 s/s]; s7-user-wd: 21s [-0.00 s/s] [01:08:44] nacht ts [01:11:14] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 64849 MB (6% inode=99%): [01:11:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:28:34] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2092.000000 [01:33:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12110.000000 [01:35:24] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (246428 kB) free! [01:35:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77583 MB (12% inode=99%): [01:36:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:36:04] aliasd on mayapple is CRITICAL: Connection refused [01:36:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:36:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:36:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 159664 MB (2% inode=68%): [01:36:34] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:36:34] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:36:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:37:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:39:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:42:49] Free Memory on damiana is OK: OK - 91.9% (7702764 kB) free. [01:56:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3640.000000 [02:06:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8462 [02:07:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:07:54] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8502.000000 [02:11:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:33:12] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11880.000000 [02:36:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77587 MB (12% inode=99%): [02:36:12] aliasd on mayapple is CRITICAL: Connection refused [02:36:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:36:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:36:54] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:36:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:37:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 159150 MB (2% inode=68%): [02:38:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:39:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:56:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4437.000000 [03:07:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:07:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10804.000000 [03:07:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10807 [03:12:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:33:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9271.000000 [03:36:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77135 MB (12% inode=99%): [03:36:44] aliasd on mayapple is CRITICAL: Connection refused [03:36:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:36:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:37:06] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:37:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 158931 MB (2% inode=68%): [03:38:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:40:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:57:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4227.000000 [04:07:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8152.000000 [04:08:06] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8149 [04:12:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:33:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:35:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8501.000000 [04:36:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77038 MB (12% inode=99%): [04:36:45] aliasd on mayapple is CRITICAL: Connection refused [04:36:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:36:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:37:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:37:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:37:09] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [04:37:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:37:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 158545 MB (2% inode=68%): [04:38:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:41:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:57:05] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3552.000000 [04:57:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5295.000000 [04:57:05] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3537 [05:10:06] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1771.000000 [05:10:06] MySQL slave on z-dat-s5-b is OK: Uptime: 101635 Threads: 4 Questions: 104799896 Slow queries: 1309 Opens: 6447 Flush tables: 1 Open tables: 256 Queries per second avg: 1031.139 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1761 [05:13:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:28:05] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2183.000000 [05:28:05] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2187 [05:33:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:35:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4988.000000 [05:36:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76927 MB (12% inode=99%): [05:36:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:37:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:37:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:37:06] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:37:45] aliasd on mayapple is CRITICAL: Connection refused [05:37:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:38:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:38:45] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 158156 MB (2% inode=68%): [05:41:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:57:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5209.000000 [06:13:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:18:45] Free Memory on turnera is WARNING: WARNING - 5.1% (430204 kB) free! [06:19:45] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (390432 kB) free! [06:28:05] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2357.000000 [06:28:05] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2356 [06:33:06] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3591.000000 [06:33:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:36:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76659 MB (12% inode=99%): [06:36:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7030.000000 [06:37:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:37:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:37:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:37:12] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:37:45] aliasd on mayapple is CRITICAL: Connection refused [06:37:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:38:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 157764 MB (2% inode=68%): [06:42:27] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1891 [06:43:42] Free Memory on turnera is OK: OK - 87.1% (7300680 kB) free. [06:47:03] Free Memory on turnera is OK: OK - 86.5% (7246472 kB) free. [06:54:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000 [06:55:43] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2165.000000 [07:02:32] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3547.000000 [07:05:23] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2084.000000 [07:05:53] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2089 [07:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:16:33] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1772.000000 [07:30:23] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1752.000000 [07:30:53] MySQL slave on daphne is OK: Uptime: 5564139 Threads: 32 Questions: 1340781193 Slow queries: 130022 Opens: 258590 Flush tables: 1 Open tables: 1591 Queries per second avg: 240.968 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1652 [07:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76461 MB (12% inode=99%): [07:37:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7063.000000 [07:37:23] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:37:54] aliasd on mayapple is CRITICAL: Connection refused [07:38:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:39:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 157033 MB (2% inode=68%): [07:46:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:53:03] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2088.000000 [07:53:33] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2107 [07:54:13] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3575.000000 [07:54:43] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1119.000000 [08:00:13] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1566.000000 [08:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:23:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3621 [08:24:04] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3644.000000 [08:30:33] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2124.000000 [08:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76284 MB (12% inode=99%): [08:37:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:37:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:37:53] aliasd on mayapple is CRITICAL: Connection refused [08:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:38:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:39:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 156630 MB (2% inode=68%): [08:44:33] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3557 [08:45:03] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3504.000000 [08:46:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3634.000000 [08:58:33] MySQL slave on z-dat-s5-b is OK: Uptime: 115348 Threads: 6 Questions: 186495194 Slow queries: 1658 Opens: 7093 Flush tables: 1 Open tables: 256 Queries per second avg: 1616.804 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [08:59:07] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1737.000000 [09:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:13:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76091 MB (12% inode=99%): [09:37:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:37:27] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:53] aliasd on mayapple is CRITICAL: Connection refused [09:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:38:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:39:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 156175 MB (2% inode=68%): [09:46:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6736.000000 [10:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75826 MB (12% inode=99%): [10:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:37:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:53] aliasd on mayapple is CRITICAL: Connection refused [10:38:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:38:23] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:39:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 155679 MB (2% inode=68%): [10:46:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9596.000000 [11:04:24] hi all [11:07:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:08:34] so due to a research project i need to set up an own instance of dewiki and probably enwiki. can somebody give me some pointers what hardware i will need? it doesnt have to run very fast - i only need to be able to make queries [11:08:51] SQL queries? [11:09:07] i will also need the whole history. so ...pages-meta-history.xml.7z [11:09:14] yes - SQL queries [11:09:29] Right, that was my next question, whether you needed all revisions or just current revisions. [11:09:33] It's not some small task, you know. [11:09:39] i know ;) [11:09:45] i don't expect it to be [11:09:59] Okay. [11:10:30] especially enwiki is rather a huge task i suppose ;) [11:10:31] Just saying, it sounds a bit naive to just come on IRC and say "I want to duplicate this massive database that's been built up for the past decade." [11:10:37] de.wiki is pretty big too. [11:10:40] i know [11:11:00] Looks like de.wikipedia is about 120 million revisions. [11:11:09] no - yeah. i hear you. but still i need to replicate it :) [11:11:19] http://www.brianstempin.com/2012/06/29/loading-the-english-wikipedia-dump-is-a-huge-pain/ might be a good start. [11:11:29] en.wikipedia is about 597 million revisions. [11:11:44] nutz: You can get a proper answer, but you can't get one here. [11:11:50] nutz: You're familiar with wikitech-l? [11:12:05] That would be your best shot at a proper answer. [11:12:27] It's also sort of within the remit of labs-l and wikitext-l. [11:12:34] But wikitech-l is the umbrella tech list. [11:12:37] ah cool [11:12:39] You happen to be in luck, kind of. [11:12:46] because? :) [11:12:52] The Wikimedia Foundation is about to do exactly what you're trying to do, kind of. [11:13:05] huh :) [11:13:06] The Wikimedia Foundation is setting up its own Toolserver, with DB replication support. [11:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:13:14] It's called Wikimedia Labs. [11:13:25] So they'll know the hardware answer, as they just provisioned this like last year. [11:13:43] Though they're obviously doing constant replication. And they're doing all Wikimedia wikis. [11:13:51] But they still probably have a decent idea. [11:13:58] That sounds very cool :) But replication support is not yet available for others? [11:14:12] Well, they provide dumps. [11:14:17] Which is more than what most sites do. ;-) [11:14:32] And I suppose that's SQL dumps, rather than freakin xml-dumps? :) [11:14:33] There's also a robust Squid cache layer. [11:14:42] There are both. [11:14:43] XML and SQL. [11:14:48] nice [11:14:52] Though both are somewhat incomplete. [11:14:56] Depends what your needs are. [11:15:23] How are they incomplete? [11:15:55] Well, the XML dumps require rebuilding the database tables after import. [11:16:15] I'd rather use SQL anyways ;) [11:16:22] Or do they have a catch too? [11:16:23] The SQL dumps don't include certain tables. [11:16:26] oh hm [11:16:33] Private tables, mostly. [11:16:47] http://dumps.wikimedia.org/ [11:16:50] What I need is mainly revisions and tables [11:17:07] http://dumps.wikimedia.org/enwiki/20130204/ [11:17:24] dumps.wikimedia.org I do know - but at least for dewiki I didn't see any sql-dumps including revisions [11:17:29] Looks like the revision table isn't available in SQL. [11:17:39] Hrm. [11:17:53] Yeah :\ [11:18:23] But to go back to my initial question. wikitech-l will be able to help, right? [11:18:42] There I will find the people who have done this recently? [11:18:49] Yes. [11:19:06] Just explain what you're working on. [11:19:08] And what you need to know. [11:19:14] And someone will likely take pity on you and answer. [11:19:19] https://lists.wikimedia.org/mailman/listinfo/wikitech-l [11:19:23] ^^ thanks [11:19:31] No problem. [11:21:09] Do you happen to know if some of these people are actually situated in Berlin? (I do prefer face-to-face over mailinglists ;) [11:27:57] there probably are [11:28:11] WMDE is rather big [11:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:59] thanks legoktm [11:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75526 MB (12% inode=99%): [11:37:22] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:54] aliasd on mayapple is CRITICAL: Connection refused [11:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:38:23] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 154067 MB (2% inode=68%): [11:46:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12452.000000 [12:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:21:43] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.336 second response time [12:22:33] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.114 second response time [12:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:03] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2148.000000 [12:36:33] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2178 [12:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74827 MB (12% inode=99%): [12:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:53] aliasd on mayapple is CRITICAL: Connection refused [12:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:38:23] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 147779 MB (2% inode=68%): [12:46:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:49:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2190 [12:52:13] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2152.000000 [12:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15179.000000 [13:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3614 [13:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3639.000000 [13:04:13] wikidata replag on z-dat-s2-b is CRITICAL: (Service Check Timed Out) [13:04:22] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2834.000000 [13:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:12:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 64427 MB (6% inode=99%): [13:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:28:33] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3132.000000 [13:29:43] MySQL slave on z-dat-s2-b is OK: Uptime: 131619 Threads: 13 Questions: 249572139 Slow queries: 2006 Opens: 1208907 Flush tables: 1 Open tables: 256 Queries per second avg: 1896.171 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1777 [13:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:43] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.735 second response time [13:36:33] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.982 second response time [13:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74547 MB (12% inode=99%): [13:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:37:43] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.074 second response time [13:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:53] aliasd on mayapple is CRITICAL: Connection refused [13:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:38:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 137412 MB (2% inode=68%): [13:41:33] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1675.000000 [13:46:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18139.000000 [14:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6891 [14:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6917.000000 [14:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:24:18] l [14:24:28] (sorry, wrong window) [14:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74315 MB (12% inode=99%): [14:37:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:53] aliasd on mayapple is CRITICAL: Connection refused [14:38:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:38:23] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 149293 MB (2% inode=68%): [14:46:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:58:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20926.000000 [15:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10226 [15:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10249.000000 [15:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74098 MB (12% inode=99%): [15:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:37:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:54] aliasd on mayapple is CRITICAL: Connection refused [15:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:38:24] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 148666 MB (2% inode=68%): [15:46:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:47:05] Tim Landscheidt * Re: [Toolserver-l] Introduction: Operators [15:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23687.000000 [16:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12307 [16:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12328.000000 [16:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:13:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:24:31] did something failed on the nightshade last reboot ? it's a dual quad-core but it remains only 4 processor after the reboot [16:27:22] errm, I mean yarrow not nightshad [16:28:26] phe: Are you sure? https://wiki.toolserver.org/view/Servers says yarrow is "2 dual-core" = 4. [16:28:45] oh, you're right [16:30:25] I'm loading way too much yarrow then, I was sure it was a 8-way box ;( [16:31:41] I checked the load a number of times, and it was always lower than nightshade, so it is my preferred box :-). [16:32:21] But I didn't take into account the number of CPUs. [16:33:10] (Both are at about 4 at the moment, but nightshade was *much* higher earlier. Perhaps the SGE enforcement for bots has helped.) [16:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73886 MB (12% inode=99%): [16:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:53] aliasd on mayapple is CRITICAL: Connection refused [16:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:38:24] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:38:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 148261 MB (2% inode=68%): [16:46:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26688.000000 [17:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14849 [17:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14854.000000 [17:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:11:53] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1877 [17:12:33] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1860.000000 [17:13:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:19:54] MySQL slave on rosemary is OK: Uptime: 594151 Threads: 13 Questions: 535312811 Slow queries: 305624 Opens: 40051 Flush tables: 1 Open tables: 3109 Queries per second avg: 900.970 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1792 [17:20:33] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1783.000000 [17:25:06] Marc A. Pelletier * [Toolserver-l] Tool Labs update and news, oh my! [17:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73673 MB (12% inode=99%): [17:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:37:53] aliasd on mayapple is CRITICAL: Connection refused [17:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:38:23] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:40:04] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 146958 MB (2% inode=68%): [17:46:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29708.000000 [18:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15193 [18:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15193.000000 [18:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:10:34] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2042.000000 [18:10:54] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2057 [18:14:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:32:06] MZMcBride * Re: [Toolserver-l] Tool Labs update and news, oh my! [18:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73458 MB (12% inode=99%): [18:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:37:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:37:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:37:54] aliasd on mayapple is CRITICAL: Connection refused [18:38:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:38:23] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 145428 MB (2% inode=68%): [18:46:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:58:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32702.000000 [19:01:06] Platonides * Re: [Toolserver-l] Tool Labs update and news, oh my! [19:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14761 [19:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14748.000000 [19:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:10:33] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2213.000000 [19:10:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:14:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:16:03] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [19:16:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [19:16:34] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [19:16:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [19:16:43] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [19:16:43] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [19:16:54] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [19:16:54] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [19:34:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73181 MB (12% inode=99%): [19:37:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:37:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:37:53] aliasd on mayapple is CRITICAL: Connection refused [19:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:38:23] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 144758 MB (2% inode=68%): [19:45:43] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2166.000000 [19:45:43] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2184.000000 [19:46:03] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2183.000000 [19:46:04] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2187.000000 [19:46:04] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2187.000000 [19:46:23] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2184.000000 [19:46:23] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2188.000000 [19:46:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:46:33] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [19:46:33] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2219.000000 [19:46:43] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2202.000000 [19:46:53] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2213.000000 [19:58:33] FMA on amaranth is OK: OK [19:58:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36207.000000 [20:02:34] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [20:03:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10539.000000 [20:06:44] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.889 second response time [20:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:07:43] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.183 second response time [20:09:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [20:09:44] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [20:10:04] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [20:10:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000 [20:10:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000 [20:10:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [20:10:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [20:10:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [20:10:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3637.000000 [20:10:33] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3660.000000 [20:10:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3643.000000 [20:10:53] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3653.000000 [20:10:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:14:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:16:03] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [20:16:34] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [20:16:34] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [20:16:43] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [20:16:44] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [20:16:54] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [20:16:54] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [20:17:36] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [20:18:43] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.799 second response time [20:20:46] Hello all [20:21:01] Did the operators already took over? ;) [20:21:05] DaB. * Re: [Toolserver-l] Introduction: Operators [20:32:11] * Merlissimo tries [20:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:34:05] ;-) [20:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72866 MB (11% inode=99%): [20:37:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:53] aliasd on mayapple is CRITICAL: Connection refused [20:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:38:33] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 143451 MB (2% inode=68%): [20:41:06] Marc A. Pelletier * Re: [Toolserver-l] Tool Labs update and news, oh my! [20:46:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:58:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39807.000000 [21:02:33] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [21:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10568.000000 [21:04:57] @replag [21:04:57] DaBPunkt: s1-rr-a: 1h 55m 5s [+1.00 s/s]; s1-rr-a-wd: 1h 55m 27s [+1.00 s/s]; s1-user: 1h 55m 5s [+1.00 s/s]; s1-user-c: 1h 55m 5s [+1.00 s/s]; s1-user-wd: 1h 55m 27s [+1.00 s/s]; s2-rr: 1h 55m 6s [+1.00 s/s]; s2-user: 1h 55m 6s [+1.00 s/s]; s2-user-c: error [21:04:58] DaBPunkt: s2-user-wd: 1h 55m 26s [+1.00 s/s]; s3-user: 1h 56m 28s [+1.00 s/s]; s3-user-wd: 1h 55m 28s [+1.00 s/s]; s4-rr-a: 1h 55m 6s [+1.00 s/s]; s4-user: 1h 55m 6s [+1.00 s/s]; s4-user-wd: 11h 9m 50s [+1.00 s/s]; s5-rr-a: 1h 55m 6s [+1.00 s/s]; s5-user: 2h 56m 54s [+0.37 s/s] [21:04:59] DaBPunkt: s5-user-c: error; s6-user: 1h 55m 6s [+1.00 s/s]; s6-user-wd: 1h 55m 28s [+1.00 s/s]; s7-user: 1h 55m 11s [+1.00 s/s]; s7-user-wd: 1h 55m 28s [+1.00 s/s] [21:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:09:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7207.000000 [21:09:44] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7224.000000 [21:10:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7223.000000 [21:10:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7226.000000 [21:10:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7227.000000 [21:10:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6874.000000 [21:10:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7014.000000 [21:10:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7255.000000 [21:10:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7194.000000 [21:10:33] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7260.000000 [21:10:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6650.000000 [21:10:53] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4488.000000 [21:10:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7208 [21:11:53] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3350.000000 [21:12:03] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3108 [21:13:53] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 803.000000 [21:14:03] MySQL slave on cassia is OK: Uptime: 954667 Threads: 9 Questions: 1041383122 Slow queries: 20952 Opens: 83252 Flush tables: 2 Open tables: 14064 Queries per second avg: 1090.833 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 389 [21:14:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:14:32] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3453 [21:16:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3621 [21:16:44] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6110 [21:16:44] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4193 [21:17:33] MySQL slave on z-dat-s4-a is OK: Uptime: 4140312 Threads: 1 Questions: 433694226 Slow queries: 234 Opens: 667 Flush tables: 11 Open tables: 222 Queries per second avg: 104.749 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1461 [21:17:33] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4252 [21:17:33] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2909 [21:17:54] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6324 [21:17:54] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6432 [21:18:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3404 [21:19:32] MySQL slave on z-dat-s6-a is OK: Uptime: 2019413 Threads: 7 Questions: 814328806 Slow queries: 103679 Opens: 3588204 Flush tables: 1 Open tables: 3241 Queries per second avg: 403.250 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1282 [21:20:33] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3288 [21:22:43] MySQL slave on z-dat-s2-b is OK: Uptime: 159999 Threads: 9 Questions: 291221567 Slow queries: 2386 Opens: 1473488 Flush tables: 1 Open tables: 256 Queries per second avg: 1820.146 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1678 [21:25:33] MySQL slave on z-dat-s7-a is OK: Uptime: 2019773 Threads: 4 Questions: 833315469 Slow queries: 39939 Opens: 2993573 Flush tables: 1 Open tables: 3974 Queries per second avg: 412.578 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1418 [21:30:44] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3590.000000 [21:30:44] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3472 [21:34:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72551 MB (11% inode=99%): [21:37:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:37:43] MySQL slave on z-dat-s3-a is OK: Uptime: 157100 Threads: 37 Questions: 214216310 Slow queries: 8110 Opens: 1939437 Flush tables: 1 Open tables: 16384 Queries per second avg: 1363.566 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1719 [21:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:37:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:37:53] aliasd on mayapple is CRITICAL: Connection refused [21:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:38:33] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:40:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 142599 MB (2% inode=68%): [21:40:43] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1766.000000 [21:42:53] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3536 [21:43:24] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3491.000000 [21:45:24] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3581.000000 [21:45:53] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3547 [21:46:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:48:05] Dr. Trigon * [Toolserver-l] mayapple config & cron jobs [21:55:22] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1732.000000 [21:55:54] MySQL slave on thyme is OK: Uptime: 597420 Threads: 12 Questions: 414038757 Slow queries: 47292 Opens: 3659 Flush tables: 1 Open tables: 457 Queries per second avg: 693.44 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1637 [21:58:22] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1763.000000 [21:58:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43021.000000 [21:58:53] MySQL slave on daphne is OK: Uptime: 5616220 Threads: 53 Questions: 1345861203 Slow queries: 140494 Opens: 260543 Flush tables: 1 Open tables: 1610 Queries per second avg: 239.638 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1631 [22:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12581 [22:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12599.000000 [22:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:09:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8743.000000 [22:09:43] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3515.000000 [22:10:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6494.000000 [22:10:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7422.000000 [22:10:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7363.000000 [22:10:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9631.000000 [22:10:34] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7026.000000 [22:10:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8278.000000 [22:10:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8275 [22:14:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:16:43] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1491.000000 [22:34:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:03] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3577.000000 [22:36:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72227 MB (11% inode=99%): [22:37:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:37:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:37:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:54] aliasd on mayapple is CRITICAL: Connection refused [22:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:38:33] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:38:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:40:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 141487 MB (2% inode=68%): [22:46:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:48:33] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3576.000000 [22:49:03] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1692.000000 [22:58:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45926.000000 [23:02:03] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3595.000000 [23:02:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15136 [23:03:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15108.000000 [23:05:33] wikidata replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1763.000000 [23:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:09:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7481.000000 [23:10:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4362.000000 [23:10:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7161.000000 [23:10:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8422.000000 [23:10:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8432