[00:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 871880.000000 [00:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:18:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31942.000000 [00:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 538621 MB (10% inode=68%): [00:28:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:34:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:34:54] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:37:02] Free Memory on damiana is WARNING: WARNING - 5.2% (438932 kB) free! [00:37:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:40:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422664 kB) free! [00:42:27] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74861 MB (12% inode=99%): [00:43:02] Free Memory on damiana is WARNING: WARNING - 5.1% (424452 kB) free! [00:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:53:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91247 [00:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91440.000000 [01:20:54] Free Memory on damiana is WARNING: WARNING - 5.4% (455212 kB) free! [01:22:12] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 57773 MB (11% inode=99%): [01:42:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74596 MB (12% inode=99%): [01:43:02] Free Memory on damiana is WARNING: WARNING - 5.3% (439944 kB) free! [01:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:49:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (406300 kB) free! [01:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90500 [01:53:33] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90061.000000 [02:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 866784.000000 [02:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:11:03] Free Memory on damiana is WARNING: WARNING - 5.1% (427816 kB) free! [02:18:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33475.000000 [02:22:41] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 540116 MB (10% inode=68%): [02:28:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:38:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:42:26] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74223 MB (12% inode=99%): [02:46:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:48:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [02:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88090 [02:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88092.000000 [02:59:03] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420216 kB) free! [03:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 864016.000000 [03:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:11:02] Free Memory on damiana is OK: OK - 9.1% (760640 kB) free. [03:19:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32980.000000 [03:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 540635 MB (10% inode=68%): [03:28:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:29:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:29:03] Free Memory on damiana is WARNING: WARNING - 5.7% (474792 kB) free! [03:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:37:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:38:02] Free Memory on damiana is CRITICAL: CRITICAL - 3.7% (306540 kB) free! [03:42:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73960 MB (12% inode=99%): [03:44:13] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 58351 MB (11% inode=99%): [03:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:47:02] Free Memory on damiana is WARNING: WARNING - 5.2% (439456 kB) free! [03:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:51:12] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 57013 MB (10% inode=99%): [03:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87897 [03:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:59:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87975.000000 [03:59:02] Free Memory on damiana is OK: OK - 7.1% (592336 kB) free. [04:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 863364.000000 [04:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:08:02] Free Memory on damiana is WARNING: WARNING - 6.5% (547180 kB) free! [04:16:06] Free Memory on damiana is OK: OK - 7.3% (614432 kB) free. [04:19:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33561.000000 [04:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 539717 MB (10% inode=68%): [04:28:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73768 MB (12% inode=99%): [04:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88467 [04:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88346.000000 [05:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 848807.000000 [05:04:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:10:02] Free Memory on damiana is WARNING: WARNING - 6.7% (558432 kB) free! [05:12:42] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1897 [05:19:02] Free Memory on damiana is CRITICAL: CRITICAL - 2.5% (212892 kB) free! [05:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36062.000000 [05:21:02] Free Memory on damiana is OK: OK - 9.9% (826156 kB) free. [05:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 540701 MB (10% inode=68%): [05:28:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73565 MB (12% inode=99%): [05:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90241 [05:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:53:42] MySQL slave on z-dat-s1-b is OK: Uptime: 373996 Threads: 4 Questions: 258524895 Slow queries: 10439 Opens: 3191 Flush tables: 1 Open tables: 256 Queries per second avg: 691.250 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1772 [05:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90375.000000 [06:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 828415.000000 [06:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38069.000000 [06:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 540885 MB (10% inode=68%): [06:28:03] Free Memory on damiana is WARNING: WARNING - 6.8% (568076 kB) free! [06:28:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:35:03] Free Memory on damiana is OK: OK - 7.1% (597336 kB) free. [06:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73336 MB (12% inode=99%): [06:45:02] Free Memory on damiana is WARNING: WARNING - 6.4% (534696 kB) free! [06:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91097 [06:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:54:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (361348 kB) free! [06:56:51] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91119.000000 [07:02:03] Free Memory on damiana is WARNING: WARNING - 5.7% (479104 kB) free! [07:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 807385.000000 [07:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:15:02] Free Memory on damiana is OK: OK - 7.1% (596540 kB) free. [07:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38075.000000 [07:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 540752 MB (10% inode=68%): [07:24:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (365356 kB) free! [07:28:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:32:51] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:41:02] Free Memory on damiana is WARNING: WARNING - 5.2% (434452 kB) free! [07:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73079 MB (11% inode=99%): [07:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:46:02] Load avg. on nightshade is WARNING: WARNING - load average: 15.76, 17.35, 13.48 [07:48:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89507 [07:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:03] Free Memory on damiana is OK: OK - 7.1% (591432 kB) free. [07:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89213.000000 [08:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 789072.000000 [08:04:02] Load avg. on nightshade is OK: OK - load average: 10.04, 13.68, 14.76 [08:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:07:03] Free Memory on damiana is WARNING: WARNING - 6.5% (541600 kB) free! [08:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36783.000000 [08:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 540519 MB (10% inode=68%): [08:23:02] Load avg. on nightshade is WARNING: WARNING - load average: 15.72, 16.10, 15.06 [08:27:02] Load avg. on nightshade is OK: OK - load average: 13.68, 14.86, 14.79 [08:28:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:32:07] Free Memory on damiana is OK: OK - 7.2% (599684 kB) free. [08:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:37:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:42:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72791 MB (11% inode=99%): [08:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:46:03] Free Memory on damiana is WARNING: WARNING - 6.3% (526348 kB) free! [08:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:53:03] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422724 kB) free! [08:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86815 [08:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86779.000000 [09:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 771438.000000 [09:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:15:03] Free Memory on damiana is WARNING: WARNING - 6.5% (543972 kB) free! [09:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37854.000000 [09:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 540301 MB (10% inode=68%): [09:26:02] Load avg. on nightshade is WARNING: WARNING - load average: 15.52, 15.67, 14.33 [09:27:04] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417248 kB) free! [09:28:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:29:02] Load avg. on nightshade is OK: OK - load average: 12.90, 14.64, 14.18 [09:29:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:31:02] Free Memory on damiana is WARNING: WARNING - 5.4% (449064 kB) free! [09:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:37:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:42:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72494 MB (11% inode=99%): [09:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:46:03] Free Memory on damiana is OK: OK - 7.2% (599880 kB) free. [09:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 83842 [09:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:02] Free Memory on damiana is WARNING: WARNING - 6.0% (502448 kB) free! [09:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84051.000000 [10:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 770763.000000 [10:04:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:13:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (389652 kB) free! [10:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36561.000000 [10:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 540039 MB (10% inode=68%): [10:28:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:31:12] Free Memory on damiana is WARNING: WARNING - 5.2% (434732 kB) free! [10:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:33:02] Load avg. on nightshade is WARNING: WARNING - load average: 13.05, 16.75, 15.09 [10:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:35:02] Load avg. on nightshade is OK: OK - load average: 11.45, 14.99, 14.65 [10:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:41:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (405440 kB) free! [10:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72211 MB (11% inode=99%): [10:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82778 [10:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:56:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82982.000000 [11:01:12] Free Memory on damiana is WARNING: WARNING - 5.1% (431032 kB) free! [11:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 771771.000000 [11:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:12:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (416672 kB) free! [11:13:12] Free Memory on damiana is WARNING: WARNING - 5.1% (424968 kB) free! [11:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35375.000000 [11:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 539633 MB (10% inode=68%): [11:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71932 MB (11% inode=99%): [11:43:12] Free Memory on damiana is WARNING: WARNING - 5.3% (443988 kB) free! [11:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:48:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419240 kB) free! [11:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84379 [11:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84512.000000 [12:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 772770.000000 [12:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:19:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33611.000000 [12:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 537561 MB (10% inode=68%): [12:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:31:12] Free Memory on damiana is WARNING: WARNING - 5.4% (451200 kB) free! [12:32:53] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:41:22] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2171.000000 [12:41:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2180.000000 [12:42:02] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2182.000000 [12:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71663 MB (11% inode=99%): [12:42:32] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2208.000000 [12:43:52] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2198.000000 [12:45:12] Free Memory on damiana is OK: OK - 7.3% (607676 kB) free. [12:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86155 [12:53:31] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:54:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (387396 kB) free! [12:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85961.000000 [13:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 775503.000000 [13:03:12] Free Memory on damiana is WARNING: WARNING - 5.3% (444900 kB) free! [13:04:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 39524 MB (4% inode=99%): [13:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:05:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3611.000000 [13:05:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [13:06:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3621.000000 [13:06:32] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3648.000000 [13:07:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [13:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35228.000000 [13:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 537281 MB (10% inode=68%): [13:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:36:13] Free Memory on damiana is OK: OK - 7.1% (597120 kB) free. [13:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71436 MB (11% inode=99%): [13:43:32] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3551.000000 [13:43:52] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3501.000000 [13:44:02] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3452.000000 [13:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:48:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84463 [13:53:33] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:56:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (416400 kB) free! [13:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:32] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1752.000000 [13:57:52] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1716.000000 [13:58:02] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1700.000000 [13:58:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3565.000000 [13:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84632.000000 [14:02:12] Free Memory on damiana is WARNING: WARNING - 5.1% (425336 kB) free! [14:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 777230.000000 [14:03:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (423032 kB) free! [14:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:05:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5502.000000 [14:19:22] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3527.000000 [14:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33827.000000 [14:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 537038 MB (10% inode=68%): [14:23:12] Free Memory on damiana is WARNING: WARNING - 5.5% (462592 kB) free! [14:28:34] hello all [14:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:31:12] Free Memory on damiana is OK: OK - 7.1% (595816 kB) free. [14:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:33:08] I move sql-s1-rr to the new server for some testing [14:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:34:52] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1420.000000 [14:36:22] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1725.000000 [14:37:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:38:13] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (396148 kB) free! [14:42:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71188 MB (11% inode=99%): [14:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:48:13] Free Memory on damiana is WARNING: WARNING - 5.1% (424828 kB) free! [14:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84791 [14:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85083.000000 [15:03:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 773680.000000 [15:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:17:12] Free Memory on damiana is WARNING: WARNING - 6.2% (522028 kB) free! [15:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30614.000000 [15:22:42] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2146 [15:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 533041 MB (10% inode=68%): [15:23:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (391072 kB) free! [15:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:31:12] Free Memory on damiana is WARNING: WARNING - 5.4% (455700 kB) free! [15:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:37:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:42:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70968 MB (11% inode=99%): [15:46:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:50:43] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3617 [15:51:12] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 56196 MB (10% inode=99%): [15:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 83456 [15:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83127.000000 [16:03:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 771133.000000 [16:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:11:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (408088 kB) free! [16:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23623.000000 [16:22:05] putting sql-s1-rr back to rosemary [16:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 528206 MB (9% inode=67%): [16:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:31:12] Free Memory on damiana is WARNING: WARNING - 5.3% (441620 kB) free! [16:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). warnings detected. run with -v for information. [16:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70697 MB (11% inode=99%): [16:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:50:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5545 [16:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82075 [16:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (337172 kB) free! [16:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81856.000000 [17:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 759829.000000 [17:04:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:09:11] [[Special:Log/newusers]] create 10 * Saminmihail * (New user account) [17:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22339.000000 [17:20:01] @replag [17:20:02] Betacommand: s1-rr-a: 1h 26m 44s [+0.03 s/s]; s1-rr-a-wd: 20s [-0.01 s/s]; s1-user: 22h 42m 23s [+0.05 s/s]; s1-user-wd: 6h 12m 12s [+0.07 s/s]; s2-rr: 22s [-0.00 s/s]; s2-user: 22s [-0.00 s/s]; s2-user-c: error; s2-user-wd: 20s [-0.01 s/s] [17:20:03] Betacommand: s5-user: 5m 50s [-0.02 s/s]; s5-user-c: 1w 1d 18h 47m 48s [-1.26 s/s] [17:21:24] DaBPunkt: any ETA on getting s1 usable again without having to fight the query killer [17:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 523241 MB (9% inode=67%): [17:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:29:12] Free Memory on damiana is WARNING: WARNING - 5.1% (428912 kB) free! [17:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:41:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421292 kB) free! [17:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70441 MB (11% inode=99%): [17:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [17:50:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3659 [17:51:42] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3580 [17:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 80943 [17:53:33] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:56:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:12] Free Memory on damiana is WARNING: WARNING - 5.1% (428088 kB) free! [17:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80455.000000 [18:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 749559.000000 [18:04:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:07:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (354756 kB) free! [18:19:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17777.000000 [18:22:12] Free Memory on damiana is WARNING: WARNING - 5.2% (437080 kB) free! [18:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 518601 MB (9% inode=67%): [18:26:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (416152 kB) free! [18:28:32] / on willow is WARNING: DISK WARNING - free space: / 21244 MB (20% inode=98%): [18:28:32] /tmp on willow is WARNING: DISK WARNING - free space: / 21244 MB (20% inode=98%): [18:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:30:48] MySQL slave on z-dat-s1-b is OK: Uptime: 419418 Threads: 5 Questions: 290632022 Slow queries: 10976 Opens: 3507 Flush tables: 1 Open tables: 256 Queries per second avg: 692.941 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1727 [18:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:36:19] Free Memory on damiana is WARNING: WARNING - 5.2% (438564 kB) free! [18:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70161 MB (11% inode=99%): [18:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:48:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77630 [18:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:56:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77354.000000 [19:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743795.000000 [19:04:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:15:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (362692 kB) free! [19:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13205.000000 [19:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 513867 MB (9% inode=67%): [19:28:32] / on willow is WARNING: DISK WARNING - free space: / 14693 MB (14% inode=98%): [19:28:33] /tmp on willow is WARNING: DISK WARNING - free space: / 14693 MB (14% inode=98%): [19:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:29:14] which dns server uses the ts? [19:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:31:12] Free Memory on damiana is WARNING: WARNING - 5.8% (486248 kB) free! [19:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:42:26] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69898 MB (11% inode=99%): [19:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:48:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75875 [19:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:33] / on willow is CRITICAL: DISK CRITICAL - free space: / 11426 MB (10% inode=98%): [19:59:33] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11426 MB (10% inode=98%): [19:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75828.000000 [20:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 720473.000000 [20:04:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:08:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (369128 kB) free! [20:09:32] / on willow is WARNING: DISK WARNING - free space: / 11597 MB (11% inode=98%): [20:09:33] /tmp on willow is WARNING: DISK WARNING - free space: / 11597 MB (11% inode=98%): [20:19:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9971.000000 [20:22:32] / on willow is CRITICAL: DISK CRITICAL - free space: / 11442 MB (10% inode=98%): [20:22:32] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11442 MB (10% inode=98%): [20:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 509310 MB (9% inode=66%): [20:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:31:10] What type of documentation is necessary to create an account? [20:31:12] Free Memory on damiana is WARNING: WARNING - 6.4% (539540 kB) free! [20:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69658 MB (11% inode=99%): [20:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:49:31] What type of documentation is necessary to create an account? [20:50:34] Hi. [20:50:45] nlv: The process is documented at... [20:51:09] nlv: https://wiki.toolserver.org/view/Account_approval_process [20:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75732 [20:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:54:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (423104 kB) free! [20:55:06] What type of personal information [20:55:12] Free Memory on damiana is WARNING: WARNING - 5.6% (467116 kB) free! [20:55:47] Username and e-mail address, mostly. [20:56:11] No identification documents? [20:56:12] Free Memory on damiana is OK: OK - 7.3% (614860 kB) free. [20:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:40] Did you read the wiki page? [20:57:43] It's pretty detailed. [20:58:08] I believe you have to privately provide a real name to the Toolserver roots. [20:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75566.000000 [21:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 711701.000000 [21:04:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9061.000000 [21:22:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (347456 kB) free! [21:22:32] / on willow is CRITICAL: DISK CRITICAL - free space: / 4923 MB (4% inode=95%): [21:22:32] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 4923 MB (4% inode=95%): [21:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 504723 MB (9% inode=66%): [21:26:12] Free Memory on damiana is WARNING: WARNING - 5.1% (427252 kB) free! [21:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:34:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:34:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69370 MB (11% inode=99%): [21:44:42] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1901 [21:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:49:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75778 [21:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75873.000000 [22:02:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (358464 kB) free! [22:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 710655.000000 [22:04:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:18:12] Free Memory on damiana is WARNING: WARNING - 5.1% (425040 kB) free! [22:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9138.000000 [22:22:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (389848 kB) free! [22:22:32] / on willow is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=0%): [22:22:33] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=0%): [22:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 499792 MB (9% inode=66%): [22:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:29:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:31:12] Free Memory on damiana is WARNING: WARNING - 5.3% (441292 kB) free! [22:32:51] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69171 MB (11% inode=99%): [22:44:42] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2417 [22:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:47:12] Free Memory on damiana is OK: OK - 7.3% (610972 kB) free. [22:49:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:53:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 76282 [22:53:33] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:56:32] / on willow is OK: DISK OK - free space: / 57427 MB (54% inode=99%): [22:56:32] /tmp on willow is OK: DISK OK - free space: / 57427 MB (54% inode=99%): [22:56:51] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:58] I cleaned willow's userlog-file [22:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76395.000000 [23:01:12] Free Memory on damiana is WARNING: WARNING - 6.6% (555936 kB) free! [23:03:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 702894.000000 [23:05:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:19:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8834.000000 [23:21:12] Free Memory on damiana is OK: OK - 7.2% (603688 kB) free. [23:22:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 494220 MB (9% inode=66%): [23:29:12] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:29:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:35:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419416 kB) free! [23:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:42:12] Free Memory on damiana is WARNING: WARNING - 5.1% (425156 kB) free! [23:42:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68788 MB (11% inode=99%): [23:44:42] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3457 [23:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:53:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77174 [23:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:55:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3605 [23:56:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77245.000000