[00:01:03] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417380 kB) free! [00:02:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 841948.000000 [00:03:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:16:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3656.000000 [00:18:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3940 [00:19:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3930.000000 [00:21:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 548429 MB (10% inode=68%): [00:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:28:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:38:02] Free Memory on damiana is WARNING: WARNING - 5.1% (429124 kB) free! [00:41:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (415580 kB) free! [00:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81517 MB (13% inode=99%): [00:47:12] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 57702 MB (10% inode=99%): [00:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47653 [00:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:56:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47826.000000 [01:02:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 845125.000000 [01:03:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:04:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43134 MB (4% inode=99%): [01:05:02] Free Memory on damiana is WARNING: WARNING - 5.1% (430036 kB) free! [01:17:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5272.000000 [01:18:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5905 [01:19:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5931.000000 [01:20:12] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 57899 MB (11% inode=99%): [01:21:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 551103 MB (10% inode=68%): [01:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:27:12] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 57459 MB (10% inode=99%): [01:28:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:32:51] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81328 MB (13% inode=99%): [01:45:12] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 58481 MB (11% inode=99%): [01:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:49:02] Free Memory on damiana is CRITICAL: CRITICAL - 3.6% (305508 kB) free! [01:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47957 [01:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:56:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48156.000000 [02:02:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 848392.000000 [02:03:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:10:02] Free Memory on damiana is WARNING: WARNING - 5.1% (423848 kB) free! [02:17:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6765.000000 [02:18:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7684 [02:19:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7686.000000 [02:21:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (364304 kB) free! [02:21:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 550431 MB (10% inode=68%): [02:26:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:28:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81091 MB (13% inode=99%): [02:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [02:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48364 [02:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:56:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48580.000000 [03:01:33] Eerggehurchtt. [03:01:50] @replag [03:01:51] ITAC: s1-rr-a-wd: 14s [-0.02 s/s]; s1-user: 13h 32m 24s [+0.23 s/s]; s1-user-wd: 1h 53m 58s [+0.13 s/s]; s2-user-c: error; s2-user-wd: 14s [-0.02 s/s]; s3-user: 1m 39s [+0.00 s/s]; s5-user: 1h 26m 6s [-0.11 s/s]; s5-user-c: 1w 2d 20h 15m 19s [+0.75 s/s] [03:02:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 850523.000000 [03:02:16] Hm. [03:02:21] What all does this mean? [03:03:03] Free Memory on damiana is WARNING: WARNING - 5.1% (428524 kB) free! [03:03:10] ????? [03:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:04:31] wat [03:04:42] Should I have done that? [03:07:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (412400 kB) free! [03:07:25] Um ping smomeone [03:11:02] Free Memory on damiana is OK: OK - 8.4% (704412 kB) free. [03:17:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6677.000000 [03:19:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5341.000000 [03:19:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5374 [03:21:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 550645 MB (10% inode=68%): [03:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:28:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:28:12] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 57723 MB (10% inode=99%): [03:32:13] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 57777 MB (11% inode=99%): [03:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:37:02] Free Memory on damiana is CRITICAL: CRITICAL - 2.0% (164068 kB) free! [03:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:39:13] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 56613 MB (10% inode=99%): [03:42:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80874 MB (13% inode=99%): [03:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:50:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3502.000000 [03:50:32] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3347 [03:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50807 [03:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:56:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50972.000000 [04:02:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 852636.000000 [04:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:05:32] MySQL slave on z-dat-s5-b is OK: Uptime: 791577 Threads: 7 Questions: 1261369128 Slow queries: 12086 Opens: 771550 Flush tables: 1 Open tables: 256 Queries per second avg: 1593.488 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1773 [04:06:03] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1699.000000 [04:09:13] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 57672 MB (10% inode=99%): [04:17:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8150.000000 [04:21:43] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 549470 MB (10% inode=68%): [04:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:28:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:32:53] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:38:02] Free Memory on damiana is CRITICAL: CRITICAL - 2.6% (214212 kB) free! [04:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80694 MB (13% inode=99%): [04:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [04:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 53172 [04:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:54:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2103 [04:55:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2119.000000 [04:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:58:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53384.000000 [05:02:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 855770.000000 [05:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:06:42] MySQL slave on z-dat-s5-b is OK: Uptime: 795239 Threads: 6 Questions: 1262205228 Slow queries: 12136 Opens: 771976 Flush tables: 1 Open tables: 256 Queries per second avg: 1587.202 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [05:07:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1758.000000 [05:17:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9940.000000 [05:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 550561 MB (10% inode=68%): [05:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:26:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1987 [05:27:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1987.000000 [05:28:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:38:03] Free Memory on damiana is CRITICAL: CRITICAL - 1.7% (138796 kB) free! [05:38:42] MySQL slave on z-dat-s5-b is OK: Uptime: 797159 Threads: 5 Questions: 1263375942 Slow queries: 12165 Opens: 772079 Flush tables: 1 Open tables: 255 Queries per second avg: 1584.848 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1764 [05:39:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1739.000000 [05:39:02] Free Memory on damiana is WARNING: WARNING - 6.3% (530956 kB) free! [05:40:03] Free Memory on damiana is CRITICAL: CRITICAL - 3.2% (270464 kB) free! [05:41:02] Free Memory on damiana is WARNING: WARNING - 5.6% (472728 kB) free! [05:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80569 MB (13% inode=99%): [05:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:48:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:53:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55629 [05:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:56:51] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:58:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55838.000000 [06:01:02] Free Memory on damiana is CRITICAL: CRITICAL - 2.3% (192928 kB) free! [06:02:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 858768.000000 [06:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:17:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9737.000000 [06:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 550430 MB (10% inode=68%): [06:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:29:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:39:13] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 57792 MB (11% inode=99%): [06:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80436 MB (13% inode=99%): [06:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:46:13] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 57392 MB (10% inode=99%): [06:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57558 [06:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57595.000000 [07:01:02] Free Memory on damiana is CRITICAL: CRITICAL - 3.3% (280316 kB) free! [07:02:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 857426.000000 [07:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:17:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9154.000000 [07:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 550253 MB (10% inode=68%): [07:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80264 MB (13% inode=99%): [07:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:47:02] Load avg. on nightshade is WARNING: WARNING - load average: 17.47, 15.94, 12.56 [07:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:49:03] Free Memory on damiana is WARNING: WARNING - 5.1% (423648 kB) free! [07:51:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (411820 kB) free! [07:52:02] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 32.17, 20.66, 15.11 [07:53:02] Load avg. on nightshade is WARNING: WARNING - load average: 25.36, 20.84, 15.52 [07:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 60356 [07:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60573.000000 [08:02:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 856386.000000 [08:04:02] Load avg. on nightshade is OK: OK - load average: 11.31, 14.63, 14.72 [08:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:18:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9102.000000 [08:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 549892 MB (10% inode=68%): [08:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:33:03] Free Memory on damiana is WARNING: WARNING - 5.2% (438588 kB) free! [08:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:41:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (400044 kB) free! [08:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80006 MB (13% inode=99%): [08:45:02] Load avg. on nightshade is WARNING: WARNING - load average: 23.45, 18.48, 15.87 [08:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:48:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:50:02] Load avg. on nightshade is OK: OK - load average: 7.59, 13.52, 14.59 [08:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62739 [08:53:31] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62898.000000 [09:02:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 854902.000000 [09:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:17:02] Free Memory on damiana is WARNING: WARNING - 5.1% (427000 kB) free! [09:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10009.000000 [09:22:41] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 547658 MB (10% inode=68%): [09:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:27:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420268 kB) free! [09:29:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:31:02] Free Memory on damiana is WARNING: WARNING - 5.6% (465748 kB) free! [09:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:37:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422144 kB) free! [09:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:38:03] Load avg. on nightshade is WARNING: WARNING - load average: 14.63, 16.53, 14.59 [09:39:12] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 58419 MB (11% inode=99%): [09:40:03] Load avg. on nightshade is OK: OK - load average: 10.74, 14.32, 14.00 [09:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79665 MB (13% inode=99%): [09:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:48:03] Free Memory on damiana is WARNING: WARNING - 5.1% (425260 kB) free! [09:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:53:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64492 [09:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64622.000000 [10:02:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 856294.000000 [10:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:16:02] Free Memory on damiana is WARNING: WARNING - 5.5% (464504 kB) free! [10:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9586.000000 [10:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 548322 MB (10% inode=68%): [10:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79400 MB (13% inode=99%): [10:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [10:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 65992 [10:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:56:12] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 57681 MB (10% inode=99%): [10:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:58:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66198.000000 [10:58:03] Free Memory on damiana is CRITICAL: CRITICAL - 3.3% (274416 kB) free! [11:02:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 855595.000000 [11:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:11:02] Free Memory on damiana is WARNING: WARNING - 5.2% (437072 kB) free! [11:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9941.000000 [11:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 547927 MB (10% inode=68%): [11:26:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:28:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421944 kB) free! [11:29:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:31:02] Free Memory on damiana is WARNING: WARNING - 5.1% (430644 kB) free! [11:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:38:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (423172 kB) free! [11:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79129 MB (12% inode=99%): [11:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 67301 [11:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67364.000000 [12:01:22] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2044.000000 [12:03:06] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 858561.000000 [12:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10398.000000 [12:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 547560 MB (10% inode=68%): [12:23:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2068.000000 [12:23:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2093 [12:27:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:29:46] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:38:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (408444 kB) free! [12:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78876 MB (12% inode=99%): [12:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 68987 [12:53:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [12:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69129.000000 [13:03:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 861561.000000 [13:04:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41504 MB (4% inode=99%): [13:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:08:42] MySQL slave on z-dat-s5-b is OK: Uptime: 824159 Threads: 6 Questions: 1362991119 Slow queries: 12537 Opens: 775332 Flush tables: 1 Open tables: 256 Queries per second avg: 1653.796 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1772 [13:09:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1757.000000 [13:14:06] Hi, how can I checkout other man's tool code with svn? I can browse on fisheye - https://fisheye.toolserver.org/browse/erfgoed I can checkout my repository [13:14:13] svn co https://ilya@svn.toolserver.org/svnroot/ilya [13:14:13] Checked out revision 0. [13:14:45] ilya@willow:~/wle/erfgoed$ svn co https://ilya@svn.toolserver.org/svnroot/erfgoed [13:14:46] svn: Could not open the requested SVN filesystem [13:14:46] ilya@willow:~/wle/erfgoed$ svn co https://ilya@svn.toolserver.org/svnroot/~erfgoed [13:14:47] svn: Could not open the requested SVN filesystem [13:17:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2028 [13:18:03] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2034.000000 [13:18:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10361.000000 [13:22:22] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3532.000000 [13:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 547274 MB (10% inode=68%): [13:25:32] Already found out - svn co https://svn.toolserver.org/svnroot/p_erfgoed [13:27:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:34:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:38:02] Free Memory on damiana is CRITICAL: CRITICAL - 1.8% (148100 kB) free! [13:40:42] MySQL slave on z-dat-s5-b is OK: Uptime: 826079 Threads: 5 Questions: 1363710103 Slow queries: 12560 Opens: 775499 Flush tables: 1 Open tables: 256 Queries per second avg: 1650.822 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1739 [13:41:06] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1701.000000 [13:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78629 MB (12% inode=99%): [13:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:48:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:53:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1998.000000 [13:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 70553 [13:53:33] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:54:02] Free Memory on damiana is WARNING: WARNING - 5.1% (429244 kB) free! [13:54:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1983 [13:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1791.000000 [13:59:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70704.000000 [13:59:42] MySQL slave on z-dat-s5-b is OK: Uptime: 827219 Threads: 7 Questions: 1364274559 Slow queries: 12575 Opens: 775620 Flush tables: 1 Open tables: 256 Queries per second avg: 1649.230 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1769 [14:03:04] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 863743.000000 [14:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:05:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (402424 kB) free! [14:09:12] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 58136 MB (11% inode=99%): [14:10:02] Free Memory on damiana is WARNING: WARNING - 5.3% (442924 kB) free! [14:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11619.000000 [14:22:22] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3032.000000 [14:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 546900 MB (10% inode=68%): [14:27:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:29:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:32:53] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:35:12] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 57708 MB (10% inode=99%): [14:37:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:42:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78366 MB (12% inode=99%): [14:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:51:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1978.000000 [14:51:43] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1991 [14:53:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 72355 [14:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:54:22] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1763.000000 [14:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:42] MySQL slave on z-dat-s5-b is OK: Uptime: 830759 Threads: 6 Questions: 1365734182 Slow queries: 12644 Opens: 775856 Flush tables: 1 Open tables: 256 Queries per second avg: 1643.959 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1793 [14:59:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [14:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72596.000000 [14:59:03] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417152 kB) free! [15:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 865639.000000 [15:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12777.000000 [15:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 546415 MB (10% inode=68%): [15:25:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1976.000000 [15:25:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1947 [15:27:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:28:42] MySQL slave on z-dat-s5-b is OK: Uptime: 832559 Threads: 6 Questions: 1366497746 Slow queries: 12680 Opens: 776245 Flush tables: 1 Open tables: 256 Queries per second avg: 1641.322 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1764 [15:29:03] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1745.000000 [15:29:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:31:03] Free Memory on damiana is WARNING: WARNING - 5.7% (480928 kB) free! [15:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:32:54] [[Tools outside Toolserver]] ! 10https://wiki.toolserver.org/w/index.php?diff=7884&oldid=7573&rcid=21707 * 70.179.169.158 * (+160) (/* Other */ ) [15:34:44] I thought that was spam. It's not. [15:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:41:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1880 [15:42:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1872.000000 [15:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78119 MB (12% inode=99%): [15:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:49:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [15:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 74217 [15:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:12] hello all [15:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74427.000000 [16:03:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 867694.000000 [16:03:03] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (423152 kB) free! [16:04:02] Free Memory on damiana is WARNING: WARNING - 5.1% (426592 kB) free! [16:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:08:08] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (404324 kB) free! [16:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14576.000000 [16:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 545900 MB (10% inode=68%): [16:25:03] Free Memory on damiana is WARNING: WARNING - 5.3% (440324 kB) free! [16:27:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:37:03] Free Memory on damiana is OK: OK - 7.2% (601012 kB) free. [16:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77796 MB (12% inode=99%): [16:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:47:02] Free Memory on damiana is WARNING: WARNING - 6.8% (569212 kB) free! [16:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 76700 [16:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76953.000000 [17:01:32] @replag [17:01:33] ITAC: s1-user: 21h 24m 48s [+0.50 s/s]; s1-user-c: 2m 19s [+0.00 s/s]; s1-user-wd: 4h 32m 41s [+0.33 s/s]; s2-user-c: error; s3-user: 1m 13s [-0.05 s/s]; s5-user: 31m 30s [+0.06 s/s]; s5-user-c: 1w 3d 1h 19m 3s [+0.59 s/s] [17:01:39] Woah. [17:03:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 868812.000000 [17:03:06] That's a lot for that s5-user-c. [17:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:05:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2020 [17:06:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2034.000000 [17:07:02] Free Memory on damiana is OK: OK - 7.2% (604924 kB) free. [17:17:42] MySQL slave on z-dat-s5-b is OK: Uptime: 839099 Threads: 6 Questions: 1369590596 Slow queries: 12805 Opens: 778040 Flush tables: 1 Open tables: 255 Queries per second avg: 1632.215 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1800 [17:18:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1763.000000 [17:18:02] Free Memory on damiana is WARNING: WARNING - 6.1% (514672 kB) free! [17:18:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17277.000000 [17:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 545423 MB (10% inode=68%): [17:27:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:42:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77422 MB (12% inode=99%): [17:45:02] Free Memory on damiana is CRITICAL: CRITICAL - 3.7% (311092 kB) free! [17:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 79245 [17:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:55:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1961 [17:56:03] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1969.000000 [17:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79502.000000 [18:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 870674.000000 [18:04:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:05:02] Load avg. on adenia is WARNING: WARNING - load average: 16.21, 16.07, 12.21 [18:11:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1772.000000 [18:11:42] MySQL slave on z-dat-s5-b is OK: Uptime: 842339 Threads: 5 Questions: 1371147130 Slow queries: 12851 Opens: 778325 Flush tables: 1 Open tables: 256 Queries per second avg: 1627.785 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1720 [18:12:02] Load avg. on adenia is OK: OK - load average: 10.27, 14.07, 12.89 [18:18:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19976.000000 [18:20:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2002 [18:21:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2023.000000 [18:22:43] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 544922 MB (10% inode=68%): [18:27:02] Free Memory on damiana is WARNING: WARNING - 5.1% (425236 kB) free! [18:27:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:38:42] MySQL slave on z-dat-s5-b is OK: Uptime: 843959 Threads: 4 Questions: 1371801467 Slow queries: 12898 Opens: 778513 Flush tables: 1 Open tables: 255 Queries per second avg: 1625.436 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1786 [18:39:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1755.000000 [18:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77055 MB (12% inode=99%): [18:45:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (341336 kB) free! [18:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:48:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1942.000000 [18:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:48:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1950 [18:53:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 81206 [18:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:56:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:43] MySQL slave on z-dat-s5-b is OK: Uptime: 845159 Threads: 6 Questions: 1372380691 Slow queries: 12913 Opens: 778647 Flush tables: 1 Open tables: 256 Queries per second avg: 1623.813 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1797 [18:59:03] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1759.000000 [18:59:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81396.000000 [19:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 872229.000000 [19:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:08:10] Free Memory on damiana is WARNING: WARNING - 5.3% (443300 kB) free! [19:12:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (414416 kB) free! [19:13:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1846.000000 [19:13:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1810 [19:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22470.000000 [19:22:43] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 544635 MB (10% inode=68%): [19:27:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:29:03] Free Memory on damiana is WARNING: WARNING - 5.2% (437228 kB) free! [19:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:33:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (363088 kB) free! [19:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:42:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76671 MB (12% inode=99%): [19:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:46:42] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2203 [19:47:02] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2175.000000 [19:48:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:53:03] Free Memory on damiana is OK: OK - 9.6% (801408 kB) free. [19:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 83509 [19:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:42] MySQL slave on z-dat-s5-b is OK: Uptime: 848759 Threads: 7 Questions: 1373858878 Slow queries: 12987 Opens: 778993 Flush tables: 1 Open tables: 256 Queries per second avg: 1618.667 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1716 [19:59:02] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1688.000000 [19:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83692.000000 [20:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 873822.000000 [20:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:13:02] Free Memory on damiana is CRITICAL: CRITICAL - 3.0% (249692 kB) free! [20:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24399.000000 [20:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 544037 MB (10% inode=68%): [20:26:03] Free Memory on damiana is WARNING: WARNING - 6.7% (559156 kB) free! [20:27:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:29:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:39:50] I will stop expresstrain for a moment for an performance-update [20:42:02] Free Memory on damiana is OK: OK - 7.4% (617600 kB) free. [20:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76344 MB (12% inode=99%): [20:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [20:53:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85077 [20:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:02] Free Memory on damiana is CRITICAL: CRITICAL - 3.9% (324388 kB) free! [20:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85200.000000 [21:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 876285.000000 [21:04:02] Free Memory on damiana is OK: OK - 10.4% (874980 kB) free. [21:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:12:22] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2179.000000 [21:16:33] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2180.000000 [21:16:52] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2200.000000 [21:16:52] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2200.000000 [21:16:53] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2210.000000 [21:17:02] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2213.000000 [21:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27266.000000 [21:19:02] Free Memory on damiana is WARNING: WARNING - 5.3% (443048 kB) free! [21:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 543672 MB (10% inode=68%): [21:28:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:29:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:37:52] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1756.000000 [21:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75949 MB (12% inode=99%): [21:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:46:03] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [21:46:22] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [21:46:52] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1724.000000 [21:47:52] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1722.000000 [21:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87026 [21:53:32] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1755.000000 [21:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:02] Free Memory on damiana is CRITICAL: CRITICAL - 3.4% (285888 kB) free! [21:59:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87194.000000 [22:03:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 877770.000000 [22:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29772.000000 [22:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 543412 MB (10% inode=68%): [22:23:02] Free Memory on damiana is WARNING: WARNING - 5.7% (475976 kB) free! [22:27:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:28:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (401064 kB) free! [22:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:32:02] Free Memory on damiana is WARNING: WARNING - 5.3% (443420 kB) free! [22:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75549 MB (12% inode=99%): [22:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87965 [22:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88232.000000 [23:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 875708.000000 [23:03:02] Free Memory on damiana is CRITICAL: CRITICAL - 3.5% (289756 kB) free! [23:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:23:41] nacht ts [23:27:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:32:02] Free Memory on damiana is WARNING: WARNING - 5.1% (430304 kB) free! [23:32:51] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:33:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (392936 kB) free! [23:34:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89566 [23:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:56:08] Free Memory on damiana is WARNING: WARNING - 5.1% (427664 kB) free! [23:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:58:08] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (335760 kB) free! [23:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89628.000000