[00:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 969359.000000 [00:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:01:52] Free Memory on damiana is WARNING: WARNING - 5.2% (435960 kB) free! [00:08:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420912 kB) free! [00:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 566769 MB (10% inode=69%): [00:25:00] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17999 [00:28:52] Free Memory on damiana is WARNING: WARNING - 5.7% (475852 kB) free! [00:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:32:43] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92606 MB (15% inode=99%): [00:44:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (370768 kB) free! [00:47:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22278 [00:52:52] Free Memory on damiana is OK: OK - 7.6% (633952 kB) free. [00:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:53:31] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22368.000000 [01:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 969408.000000 [01:00:52] Free Memory on damiana is CRITICAL: CRITICAL - 3.2% (269864 kB) free! [01:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:02:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45411 MB (4% inode=99%): [01:09:52] Free Memory on damiana is WARNING: WARNING - 5.1% (430700 kB) free! [01:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 567141 MB (10% inode=69%): [01:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18560 [01:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:32:43] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:33:52] Free Memory on damiana is WARNING: WARNING - 6.0% (504872 kB) free! [01:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92447 MB (15% inode=99%): [01:41:35] @replag [01:41:36] Betacommand: s1-rr-a: 5h 9m 36s [-0.01 s/s]; s1-rr-a-wd: 1m 25s [+0.00 s/s]; s1-user: 6h 10m 40s [-0.06 s/s]; s1-user-wd: 13m 59s [+0.01 s/s]; s2-rr: 30s [+0.00 s/s]; s2-user: 30s [+0.00 s/s]; s2-user-c: error; s2-user-wd: 1m 25s [+0.00 s/s] [01:41:37] Betacommand: s3-user: 1m 8s [-0.00 s/s]; s5-rr-a: 13s [-]; s5-user: 13s [+0.00 s/s]; s5-user-c: 1w 4d 1h 55m 43s [-2.22 s/s]; s7-user: 13s [-0.00 s/s] [01:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:48:52] Free Memory on damiana is OK: OK - 7.4% (617708 kB) free. [01:51:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22349 [01:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:53:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22337.000000 [01:57:52] Free Memory on damiana is WARNING: WARNING - 6.1% (509948 kB) free! [02:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 950589.000000 [02:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 564658 MB (10% inode=69%): [02:24:53] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:26:43] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18417 [02:31:41] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:36:52] Free Memory on damiana is OK: OK - 7.3% (613032 kB) free. [02:37:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92373 MB (15% inode=99%): [02:43:52] Free Memory on damiana is CRITICAL: CRITICAL - 3.4% (282732 kB) free! [02:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22906 [02:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:53:31] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22948.000000 [02:55:52] Free Memory on damiana is WARNING: WARNING - 5.4% (451820 kB) free! [03:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 920774.000000 [03:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:02:52] Free Memory on damiana is OK: OK - 7.1% (596216 kB) free. [03:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 561345 MB (10% inode=69%): [03:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17492 [03:26:52] Free Memory on damiana is WARNING: WARNING - 5.1% (429736 kB) free! [03:27:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (414504 kB) free! [03:29:52] Free Memory on damiana is WARNING: WARNING - 5.2% (434368 kB) free! [03:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92215 MB (15% inode=99%): [03:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22871 [03:53:23] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:53:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22951.000000 [03:59:52] Free Memory on damiana is WARNING: WARNING - 6.1% (510276 kB) free! [04:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 906075.000000 [04:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:01:32] Load avg. on ortelius is WARNING: WARNING - load average: 4.17, 17.27, 12.32 [04:02:32] Load avg. on ortelius is OK: OK - load average: 2.24, 14.36, 11.61 [04:03:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (406140 kB) free! [04:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:18:53] Free Memory on damiana is WARNING: WARNING - 5.9% (492616 kB) free! [04:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 558077 MB (10% inode=69%): [04:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17134 [04:30:52] Free Memory on damiana is OK: OK - 8.0% (668036 kB) free. [04:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:41:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92073 MB (15% inode=99%): [04:44:52] Free Memory on damiana is WARNING: WARNING - 5.8% (488936 kB) free! [04:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [04:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23138 [04:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:53:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:52] Free Memory on damiana is CRITICAL: CRITICAL - 3.5% (294440 kB) free! [04:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23114.000000 [05:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 875246.000000 [05:00:52] Free Memory on damiana is WARNING: WARNING - 5.6% (469276 kB) free! [05:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 554304 MB (10% inode=69%): [05:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17849 [05:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91941 MB (15% inode=99%): [05:47:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23375 [05:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23600.000000 [05:57:52] Free Memory on damiana is WARNING: WARNING - 5.8% (489172 kB) free! [06:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 856753.000000 [06:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:01:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (418148 kB) free! [06:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 550576 MB (10% inode=68%): [06:24:53] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:26:41] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16527 [06:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:32:52] Free Memory on damiana is WARNING: WARNING - 5.1% (424352 kB) free! [06:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:38:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421844 kB) free! [06:39:52] Free Memory on damiana is WARNING: WARNING - 5.5% (462496 kB) free! [06:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91759 MB (15% inode=99%): [06:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24137 [06:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24129.000000 [07:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 855080.000000 [07:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:12:52] Free Memory on damiana is CRITICAL: CRITICAL - 3.8% (320760 kB) free! [07:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 548077 MB (10% inode=68%): [07:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13098 [07:30:52] Free Memory on damiana is WARNING: WARNING - 5.7% (476396 kB) free! [07:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91569 MB (15% inode=99%): [07:46:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422588 kB) free! [07:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:50:52] Free Memory on damiana is WARNING: WARNING - 5.1% (426148 kB) free! [07:51:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23881 [07:51:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (413668 kB) free! [07:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:53:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23883.000000 [08:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 841895.000000 [08:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:52] Free Memory on damiana is WARNING: WARNING - 5.1% (425852 kB) free! [08:10:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (380808 kB) free! [08:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:20:43] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 545892 MB (10% inode=68%): [08:23:52] Free Memory on damiana is WARNING: WARNING - 5.4% (454328 kB) free! [08:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10151 [08:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:32:43] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:41:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91338 MB (15% inode=99%): [08:42:53] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420988 kB) free! [08:43:52] Free Memory on damiana is WARNING: WARNING - 5.7% (473792 kB) free! [08:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24290 [08:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24219.000000 [08:59:52] Free Memory on damiana is CRITICAL: CRITICAL - 3.4% (285372 kB) free! [09:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 823878.000000 [09:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:04:53] Free Memory on damiana is WARNING: WARNING - 5.1% (423636 kB) free! [09:10:07] Marlen Caemmerer * [Toolserver-announce] IP Renumbering of the complete Toolserver cluster [09:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 543958 MB (10% inode=68%): [09:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7198 [09:26:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (348736 kB) free! [09:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:35:53] Free Memory on damiana is WARNING: WARNING - 5.2% (434960 kB) free! [09:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91134 MB (14% inode=99%): [09:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:51:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23302 [09:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23196.000000 [10:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 824270.000000 [10:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 542574 MB (10% inode=68%): [10:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4099 [10:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:35:52] Free Memory on damiana is WARNING: WARNING - 6.0% (501896 kB) free! [10:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90928 MB (14% inode=99%): [10:41:42] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3512 [10:46:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (384700 kB) free! [10:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20727 [10:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20230.000000 [11:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 826384.000000 [11:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:02:42] MySQL slave on z-dat-s1-b is OK: Uptime: 133335 Threads: 4 Questions: 71309249 Slow queries: 6562 Opens: 3058 Flush tables: 1 Open tables: 255 Queries per second avg: 534.812 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1793 [11:03:52] Free Memory on damiana is WARNING: WARNING - 5.2% (437532 kB) free! [11:08:52] Free Memory on damiana is OK: OK - 7.3% (611876 kB) free. [11:14:42] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2125 [11:16:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:16:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (416768 kB) free! [11:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 542198 MB (10% inode=68%): [11:23:42] MySQL slave on z-dat-s1-b is OK: Uptime: 134595 Threads: 4 Questions: 72228241 Slow queries: 6618 Opens: 3058 Flush tables: 1 Open tables: 255 Queries per second avg: 536.633 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1766 [11:24:53] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:31:43] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90709 MB (14% inode=99%): [11:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:50:52] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2035.000000 [11:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19024 [11:51:32] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2056 [11:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:53:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18956.000000 [12:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 828432.000000 [12:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 541942 MB (10% inode=68%): [12:23:08] Danny B. * Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster [12:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:31:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3636 [12:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:31:52] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3651.000000 [12:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:37:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90474 MB (14% inode=99%): [12:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:49:57] Hello all [12:51:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17892 [12:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17753.000000 [12:57:08] DaB. * Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster [13:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 831631.000000 [13:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:02:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46244 MB (4% inode=99%): [13:03:15] [[Operators]] 10https://wiki.toolserver.org/w/index.php?diff=7869&oldid=7845&rcid=21691 * Dab * (+41) (/* General rights */ +ML) [13:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 541603 MB (10% inode=68%): [13:24:53] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:31:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5978 [13:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:31:51] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5995.000000 [13:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90264 MB (14% inode=99%): [13:42:32] / on willow is WARNING: DISK WARNING - free space: / 14141 MB (13% inode=98%): [13:42:32] /tmp on willow is WARNING: DISK WARNING - free space: / 14141 MB (13% inode=98%): [13:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:49:32] / on willow is CRITICAL: DISK CRITICAL - free space: / 10814 MB (10% inode=97%): [13:49:32] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 10660 MB (10% inode=97%): [13:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17961 [13:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17955.000000 [13:59:32] / on willow is WARNING: DISK WARNING - free space: / 13281 MB (12% inode=98%): [13:59:32] /tmp on willow is WARNING: DISK WARNING - free space: / 13281 MB (12% inode=98%): [14:00:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (385416 kB) free! [14:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 834746.000000 [14:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:06:32] / on willow is OK: DISK OK - free space: / 22221 MB (21% inode=98%): [14:06:32] /tmp on willow is OK: DISK OK - free space: / 22221 MB (21% inode=98%): [14:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 515097 MB (9% inode=67%): [14:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:31:31] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7854 [14:31:41] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:31:52] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7871.000000 [14:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:32:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90029 MB (14% inode=99%): [14:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17799 [14:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17793.000000 [15:00:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (384748 kB) free! [15:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 837719.000000 [15:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:02:02] Free Memory on damiana is WARNING: WARNING - 5.2% (431964 kB) free! [15:03:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (393544 kB) free! [15:10:02] Free Memory on damiana is WARNING: WARNING - 5.1% (423760 kB) free! [15:11:08] DaB. * [Toolserver-l] suhosin log lines [15:18:54] DaBPunkt: Is suhosin enabled globally? /etc/php5/conf.d/suhosin.ini is commented out for the most part, and I don't know the defaults. [15:19:13] DaBPunkt: for your question in mail: how do I use suhosin? [15:19:33] (seems it's being talked about?) [15:20:43] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 514286 MB (9% inode=67%): [15:22:40] liangent: I'm not a php-programmer [15:25:03] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:25:15] The error message would suggest that some script is trying to set the configuration variable memory_limit to -1 so a "find /home -type f -name \*.php -print0 | xargs -0r fgrep memory_limit" (untested) might shed some light. [15:25:45] scfc_de: I have no ideas how the defaults are. But AFAIS there is nothing in puppet about it [15:25:55] scfc_de: well I have it [15:26:11] I have a mediawiki in my home and I'm using it as my bot's lib [15:26:40] and I'm not sure how mediawiki interacts with memory_limit [15:29:51] liangent: than can you greo your mediawiki-folder? [15:31:00] liangent: MediaWiki has several references to memory_limit in its code, but the ways it is used and set are a bit mysterious. Maybe ask in #mediawiki what will trigger it to set it to -1. [15:31:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8553 [15:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:31:52] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8527.000000 [15:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:32:43] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:34:02] Free Memory on damiana is WARNING: WARNING - 7.0% (584812 kB) free! [15:36:02] Free Memory on damiana is OK: OK - 7.1% (591524 kB) free. [15:36:03] so any suggestion for me? [15:36:15] about what I should do now [15:37:01] liangent: see scfc_de [15:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:38:56] I'm trying to run echo ini_get( 'memory_limit' ) and it's already -1?? [15:41:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89803 MB (14% inode=99%): [15:43:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (365428 kB) free! [15:43:31] liangent: As I understand it, it's not the system-wide limit that triggers the suhosin warning, but when it is (tried to be) set. So you should ask on #mediawiki if MediaWiki does actually do this (because otherwise you aren't the culprit :-)). [15:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:47:13] ping DaBPunkt [15:47:18] pong [15:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:47:30] regarding your last mail [15:47:37] suhosin is a protector for php [15:47:57] yes, I know [15:47:58] I'm quite sure that the trigger will be: ini_set('memory_limit', -1); [15:48:34] Platonides: but ini_get('memory_limit') is already -1 for me? [15:49:32] DaBPunkt, in which log is it, /var/log/userlog ? [15:49:43] Platonides: liangent has a MediaWiki instance for his bot. [15:49:49] Platonides: /var/log/syslog [15:50:10] hmm... that's not readable by users [15:50:56] Platonides: It contains not more than I posted to the ml [15:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16836 [15:52:02] does visiting http://toolserver.org/~platonides/sandbox/suhosin.php generate those lines? [15:52:39] wait, is it on the web servers or in the normal ones? [15:52:50] looks like a problem on the Debian ones [15:53:21] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:53:37] Platonides: on the userland-servers [15:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:56] AFAIS only att the debian onces, but it could be that solaris just filter it [15:54:11] I bet the solaris ones don't use the suhosin patch [15:54:58] hehehe: sendto(0, "<9>suhosin[30058]: ALERT - script tried to disable memory_limit by setting it to a negative value -1 bytes which is not allowed (attacker 'REMOTE_ADDR not set', file 'unknown')\n", 177, 0, NULL, 0) = 177 [15:55:52] platonides@nightshade ~ $ strace -s200 -esendto php -r"ini_set('memory_limit', -1);" [15:55:52] sendto(3, "\2\0\0\0\22\0\0\0\t\0\0\0services\0", 21, MSG_NOSIGNAL, NULL, 0) = 21 [15:55:52] sendto(0, "<9>suhosin[30755]: ALERT - script tried to disable memory_limit by setting it to a negative value -1 bytes which is not allowed (attacker 'REMOTE_ADDR not set', file 'unknown')\n", 177, 0, NULL, 0) = 177 [15:56:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16649.000000 [15:57:24] Platonides: So the question remains: Is it set that way in MediaWiki? :-) The code is a bit complex. [15:57:26] but I don't see a problem with a command-line script disabling the memory limit [15:58:25] scfc_de, it may be in LocalSettings.php [15:58:34] or $wgMemoryLimit could be set to -1 [15:59:34] it should be possible to disable that warning using suhosin.memory_limit [15:59:46] although it's not clear if suhosin.memory_limit = -1 completely disables it [15:59:52] http://www.hardened-php.net/suhosin/configuration.html#suhosin.memory_limit [16:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 840024.000000 [16:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:01:07] Platonides: That should be testable :-). [16:02:37] just look if you are generating that sendto [16:02:40] Platonides: I'm using maintenance scripts extensively, including the ones shipped with mediawiki and some other ones written by myself as bot scripts [16:03:02] Free Memory on damiana is WARNING: WARNING - 6.0% (499720 kB) free! [16:04:03] I think maintenance scripts do so [16:04:10] Normally we disable the memory_limit when running admin scripts. [16:04:10] * Some scripts may wish to actually set a limit, however, to avoid [16:04:10] * blowing up unexpectedly. We also support a --memory-limit option, [16:04:10] * to allow sysadmins to explicitly set one if they'd prefer to override [16:04:10] * defaults (or for people using Suhosin which yells at you for trying [16:04:11] * to disable the limits) [16:04:23] from Maintenance::memoryLimit() [16:06:35] I've never used the --memory-limit option [16:06:54] Do we have an ETA for s1 to be back to normal? [16:07:13] ah "Normally we disable the memory_limit when running admin scripts." so this is the default? [16:07:44] by default it disables the memory limit [16:07:51] hmm it's true [16:08:08] a maintenance script was run by a human [16:08:25] why limit its memory? [16:08:26] so it's my script who's generating those warnings [16:08:35] probably [16:10:17] Betacommand: only globalimagelinks is left for import [16:13:03] Platonides: I have memory limit set on SGE submission call, so I don't think disabling a memory limit in php is bad .. is this correct? [16:13:21] no, reading the suhosing source, it's not possible to disable that warning [16:13:31] no, it's not bad [16:13:40] suhosin is being paranoid [16:13:55] what if liangent just set a high limit? [16:15:44] liangent: just to be clear: If we find not solution I have to live with the loglines – there is no reason to disable a tool [16:18:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (407796 kB) free! [16:20:43] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 513836 MB (9% inode=67%): [16:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:25:53] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:31:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9123 [16:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:31:52] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9141.000000 [16:32:02] Free Memory on damiana is WARNING: WARNING - 5.7% (477864 kB) free! [16:32:43] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:32:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:37:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89569 MB (14% inode=99%): [16:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:51:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15189 [16:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15147.000000 [17:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 842580.000000 [17:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:06:00] DaBPunkt, you could remove the php5-suhosin package [17:06:16] another option would be that liangent commented the lines changing the memory limit [17:13:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420312 kB) free! [17:14:03] Free Memory on damiana is WARNING: WARNING - 5.1% (423396 kB) free! [17:20:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (409428 kB) free! [17:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 513198 MB (9% inode=67%): [17:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:25:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:31:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10014 [17:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:31:52] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10035.000000 [17:33:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:33:43] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:41:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89373 MB (14% inode=99%): [17:43:02] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (243792 kB) free! [17:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:51:03] Free Memory on damiana is WARNING: WARNING - 5.4% (451220 kB) free! [17:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13882 [17:53:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (332284 kB) free! [17:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:53:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13769.000000 [18:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 844820.000000 [18:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:08:03] Free Memory on damiana is WARNING: WARNING - 5.2% (432112 kB) free! [18:11:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419812 kB) free! [18:16:02] Free Memory on damiana is WARNING: WARNING - 5.2% (435320 kB) free! [18:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 512904 MB (9% inode=67%): [18:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:25:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:31:31] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8785 [18:31:53] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:31:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8792.000000 [18:33:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:33:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89183 MB (14% inode=99%): [18:46:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14155 [18:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:53:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14115.000000 [19:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 847269.000000 [19:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:06:07] John * [Toolserver-l] Tool request [19:08:03] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1889.000000 [19:13:07] Magnus Manske * Re: [Toolserver-l] Tool request [19:15:08] John * Re: [Toolserver-l] Tool request [19:16:03] Free Memory on damiana is WARNING: WARNING - 5.7% (481012 kB) free! [19:19:02] aliasd on mayapple is CRITICAL: Connection refused [19:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 512524 MB (9% inode=67%): [19:22:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421464 kB) free! [19:22:02] aliasd on mayapple is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [19:23:03] Free Memory on damiana is WARNING: WARNING - 5.1% (428720 kB) free! [19:23:03] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1772.000000 [19:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:25:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:28:03] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (378356 kB) free! [19:31:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8601 [19:31:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:31:52] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8622.000000 [19:33:07] Maarten Dammers * Re: [Toolserver-l] [Maps-l] IP Renumbering of the complete Toolserver cluster [19:33:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:33:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:39:03] Free Memory on damiana is WARNING: WARNING - 5.2% (436324 kB) free! [19:41:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88885 MB (14% inode=99%): [19:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14848 [19:53:23] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:53:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15031.000000 [20:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 848871.000000 [20:01:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:20:51] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 512479 MB (9% inode=67%): [20:25:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:25:17] [[Special:Log/newusers]] create 10 * Retusa * (New user account) [20:26:04] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:31:33] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9567 [20:31:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:31:52] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9577.000000 [20:33:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:33:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:37:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:41:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88626 MB (14% inode=99%): [20:47:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:51:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15364 [20:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:54:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15594.000000 [21:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 850726.000000 [21:00:07] Theopolisme * Re: [Toolserver-l] Tool request [21:01:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:16:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:21:43] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 512010 MB (9% inode=67%): [21:26:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:27:04] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:31:53] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:32:04] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10418.000000 [21:32:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10443 [21:33:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:33:53] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:37:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:40:33] Load avg. on ortelius is WARNING: WARNING - load average: 21.30, 19.05, 13.82 [21:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88418 MB (14% inode=99%): [21:43:33] Load avg. on ortelius is OK: OK - load average: 10.03, 14.88, 13.09 [21:47:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:51:14] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16298 [21:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:55:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16294.000000 [22:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 853051.000000 [22:00:03] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1812.000000 [22:01:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:06:03] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1793.000000 [22:21:45] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 511713 MB (9% inode=67%): [22:26:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:28:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:32:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9411 [22:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:33:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9436.000000 [22:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88120 MB (14% inode=99%): [22:45:02] Free Memory on damiana is WARNING: WARNING - 6.4% (536616 kB) free! [22:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:52:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16721 [22:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:56:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16737.000000 [23:01:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 855481.000000 [23:02:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:21:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 509833 MB (9% inode=67%): [23:25:02] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (418284 kB) free! [23:26:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:26:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 14.72, 29.14, 19.21 [23:28:02] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:28:32] Load avg. on ortelius is WARNING: WARNING - load average: 10.29, 22.60, 17.95 [23:31:03] Free Memory on damiana is WARNING: WARNING - 5.1% (425720 kB) free! [23:32:32] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9885 [23:32:52] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:33:03] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9902.000000 [23:33:03] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422196 kB) free! [23:34:32] Load avg. on ortelius is OK: OK - load average: 6.62, 12.26, 14.75 [23:34:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:34:52] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87793 MB (14% inode=99%): [23:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:48:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:52:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17067 [23:53:02] Free Memory on damiana is WARNING: WARNING - 6.9% (575880 kB) free! [23:53:32] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:56:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:58:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17049.000000