[00:25:50] nacht ts [00:30:02] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14444.000000 [00:30:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8569.000000 [00:30:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9004.000000 [00:30:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8537.000000 [00:30:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13400.000000 [00:31:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17923.000000 [00:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [00:38:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28806 [00:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:46:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47592 MB (7% inode=99%): [00:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9833 [00:51:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9845.000000 [00:53:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1286258.000000 [01:02:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49186 MB (5% inode=99%): [01:30:02] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13263.000000 [01:30:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4294.000000 [01:30:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3653.000000 [01:30:42] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3598.000000 [01:30:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13504.000000 [01:31:02] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3583.000000 [01:31:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19320.000000 [01:33:31] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [01:38:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27880 [01:39:03] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3582.000000 [01:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:47:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46426 MB (7% inode=99%): [01:47:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10891 [01:50:42] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1749.000000 [01:51:02] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1715.000000 [01:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:52:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10832.000000 [01:53:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1286228.000000 [01:58:03] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1753.000000 [02:30:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13301.000000 [02:30:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13688.000000 [02:31:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18853.000000 [02:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [02:38:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26454 [02:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:47:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47378 MB (7% inode=99%): [02:47:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:49:01] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10658 [02:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10504.000000 [02:53:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1287696.000000 [03:30:02] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12027.000000 [03:30:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11921.000000 [03:31:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18265.000000 [03:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [03:38:24] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25659 [03:46:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:47:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47236 MB (7% inode=99%): [03:47:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:47:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:49:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10002 [03:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10094.000000 [03:53:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:56:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1288948.000000 [04:30:02] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8385.000000 [04:30:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12357.000000 [04:31:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18638.000000 [04:34:07] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [04:39:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24407 [04:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:47:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47096 MB (7% inode=99%): [04:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:47:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12175 [04:52:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12295.000000 [04:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1290349.000000 [05:25:02] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3539.000000 [05:30:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7413.000000 [05:31:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16588.000000 [05:33:31] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [05:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23629 [05:40:02] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1673.000000 [05:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:47:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46880 MB (7% inode=99%): [05:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:49:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11939 [05:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12019.000000 [05:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1280807.000000 [06:17:52] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3594.000000 [06:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:31:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12841.000000 [06:33:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [06:33:52] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1734.000000 [06:34:53] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 17.32, 26.06, 22.10 [06:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20724 [06:44:52] Load avg. on nightshade is WARNING: WARNING - load average: 17.59, 18.60, 19.94 [06:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:47:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46739 MB (7% inode=99%): [06:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:48:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:49:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12333 [06:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12139.000000 [06:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1266475.000000 [07:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:31:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6839.000000 [07:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [07:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22339 [07:44:52] Load avg. on nightshade is WARNING: WARNING - load average: 18.17, 18.83, 17.63 [07:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:47:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46571 MB (7% inode=99%): [07:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11419 [07:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11616.000000 [07:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:53:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1259824.000000 [07:59:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3512.000000 [08:17:03] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1700.000000 [08:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21362 [08:44:52] Load avg. on nightshade is WARNING: WARNING - load average: 14.84, 15.30, 16.39 [08:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:47:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46378 MB (7% inode=99%): [08:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10924 [08:52:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10751.000000 [08:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1258000.000000 [08:57:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:13:21] Load avg. on nightshade is OK: OK - load average: 13.21, 14.54, 14.99 [09:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:26:52] Load avg. on nightshade is WARNING: WARNING - load average: 18.99, 17.57, 16.10 [09:33:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:38:53] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1991.000000 [09:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18240 [09:39:32] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1922 [09:39:52] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2130.000000 [09:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106141 MB (17% inode=99%): [09:40:53] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1794.000000 [09:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:47:52] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1899.000000 [09:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:49:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10127 [09:49:32] MySQL slave on z-dat-s5-b is OK: Uptime: 380210 Threads: 9 Questions: 721503984 Slow queries: 4615 Opens: 422085 Flush tables: 1 Open tables: 255 Queries per second avg: 1897.645 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1764 [09:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10048.000000 [09:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1259332.000000 [09:58:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:20:17] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [10:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:24:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2095.000000 [10:26:52] Load avg. on nightshade is WARNING: WARNING - load average: 22.79, 20.24, 18.20 [10:32:52] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 36.46, 25.67, 20.99 [10:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16033 [10:40:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105927 MB (17% inode=99%): [10:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:46:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:48:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9484 [10:50:52] Load avg. on nightshade is WARNING: WARNING - load average: 15.82, 17.44, 19.94 [10:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9280.000000 [10:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1259772.000000 [10:58:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:03:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [11:04:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3548.000000 [11:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [11:15:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6686.000000 [11:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [11:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12856 [11:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105710 MB (17% inode=99%): [11:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:49:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10954 [11:50:52] Load avg. on nightshade is WARNING: WARNING - load average: 17.38, 16.69, 16.86 [11:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11137.000000 [11:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1255513.000000 [11:58:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3865.000000 [12:15:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5105.000000 [12:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:33:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [12:38:52] Load avg. on nightshade is OK: OK - load average: 12.42, 14.44, 14.95 [12:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9550 [12:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105506 MB (17% inode=99%): [12:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:49:52] Load avg. on nightshade is WARNING: WARNING - load average: 13.63, 15.37, 15.22 [12:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12072 [12:52:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12223.000000 [12:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:53] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 31.01, 22.31, 18.12 [12:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1252577.000000 [12:58:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:02:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48184 MB (4% inode=99%): [13:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5960.000000 [13:15:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3804.000000 [13:16:52] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 30.02, 30.48, 27.44 [13:21:05] Federico Leva (Nemo) * [Toolserver-l] [TS logo] Fwd: Free as in Wikimedia Foundation [13:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:33:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8795 [13:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105325 MB (17% inode=99%): [13:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:46:37] hello all [13:47:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:48:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12372 [13:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12374.000000 [13:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:09] DaBPunkt: WMDE NDA is public? [13:55:05] fale: no idea [13:55:24] DaBPunkt: who can I ask to? [13:55:36] silke_meyer [13:55:44] DaBPunkt: htnkas :) [13:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1252196.000000 [13:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:03:36] @replag [14:03:37] Matthew_: s1-rr-a: error; s1-user: 3h 31m 50s [-0.17 s/s]; s1-user-c: 1h 16m 58s [-]; s1-user-wd: 2h 21m 42s [-]; s2-user-c: error; s5-user: 16m 1s [-]; s5-user-c: 2w 11h 52m 20s [-0.14 s/s] [14:03:46] ... [14:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8659.000000 [14:09:33] @replag [14:09:34] DaBPunkt: s1-rr-a: 2h 19m 55s [-]; s1-user: 3h 36m 56s [-]; s1-user-c: 1h 19m 1s [-]; s1-user-wd: 2h 26m 28s [-]; s2-user-c: error; s3-user: 23s [-]; s5-user: 4m 45s [-]; s5-user-c: 2w 11h 55m 28s [-] [14:09:43] stupid spelling error… [14:10:10] (s1-rr-a is not in rotation at the moment) [14:15:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4918.000000 [14:16:52] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 14.10, 19.27, 22.90 [14:18:53] (s1-rr-a is not in rotation at the moment) [14:18:55] @replag [14:18:57] DaBPunkt: s1-rr-a: 2h 12m 24s [-0.80 s/s]; s1-rr-a-wd: 6m 45s [-]; s1-user: 3h 44m 11s [+0.77 s/s]; s1-user-c: 1h 22m 7s [+0.33 s/s]; s1-user-wd: 2h 33m 49s [+0.78 s/s]; s2-rr: 6m 46s [-]; s2-user: 6m 46s [-]; s2-user-c: error [14:18:58] DaBPunkt: s2-user-wd: 6m 47s [-]; s5-user-c: 2w 12h 1m 17s [+0.62 s/s] [14:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:23:52] Load avg. on nightshade is WARNING: WARNING - load average: 13.34, 15.16, 19.63 [14:31:53] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 26.92, 20.94, 20.13 [14:33:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:39:33] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7079 [14:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105054 MB (17% inode=99%): [14:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:46:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14848 [14:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14984.000000 [14:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:53:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1253607.000000 [14:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:07:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11147.000000 [15:15:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4689.000000 [15:21:43] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:25:53] Load avg. on nightshade is WARNING: WARNING - load average: 8.47, 9.93, 19.82 [15:33:42] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9106 [15:34:52] Load avg. on nightshade is OK: OK - load average: 8.68, 8.95, 14.93 [15:34:52] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3590.000000 [15:36:05] Daniel Schwen * Re: [Toolserver-l] [TS logo] Fwd: Free as in Wikimedia Foundation [15:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4570 [15:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104828 MB (17% inode=99%): [15:40:42] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3565 [15:43:42] MySQL slave on z-dat-s4-a is OK: Uptime: 5416274 Threads: 2 Questions: 519214865 Slow queries: 322 Opens: 670 Flush tables: 11 Open tables: 225 Queries per second avg: 95.862 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 960 [15:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:46:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:48:04] DaB. * Re: [Toolserver-l] [TS logo] Fwd: Free as in Wikimedia Foundation [15:48:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15636 [15:52:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15767.000000 [15:53:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1255263.000000 [16:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12725.000000 [16:11:02] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2143.000000 [16:18:52] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1741.000000 [16:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:33:05] Marc A. Pelletier * Re: [Toolserver-l] [TS logo] Fwd: Free as in Wikimedia Foundation [16:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5722 [16:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104617 MB (17% inode=99%): [16:45:23] Load avg. on ortelius is WARNING: WARNING - load average: 9.03, 16.91, 13.82 [16:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:46:22] Load avg. on ortelius is OK: OK - load average: 7.41, 14.98, 13.33 [16:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16190 [16:52:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16119.000000 [16:53:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:03] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1792.000000 [16:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1256038.000000 [17:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14038.000000 [17:21:43] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:32:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2134.000000 [17:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6885 [17:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104358 MB (17% inode=99%): [17:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:46:53] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1670.000000 [17:47:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16117 [17:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15883.000000 [17:53:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:53:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1256114.000000 [18:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:00:05] Sudhakara alva * Re: [Toolserver-l] [TS logo] Fwd: Free as in Wikimedia Foundation [18:04:05] DaB. * [Toolserver-l] Status of the s1-reimport on sql-s1-rr [18:05:18] Is it just me or is Sudhakara's mail without a reply? [18:06:36] There is no content indeed. [18:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14310.000000 [18:07:20] ok. [18:07:29] I have to levae for dinner. cu [18:07:59] Bon appétit. [18:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6574 [18:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104144 MB (17% inode=99%): [18:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:46:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:48:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:50:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15413 [18:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15563.000000 [18:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1256809.000000 [19:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:07:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13552.000000 [19:13:42] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:14:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:16:32] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:16:52] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:17:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:17:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:20:02] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2065.000000 [19:20:43] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:21:16] re [19:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:22:42] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2109 [19:23:32] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:39:34] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6097 [19:40:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103863 MB (17% inode=99%): [19:40:52] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:42:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:50:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17401 [19:52:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [19:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17450.000000 [19:53:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:05] DaB. * [Toolserver-announce] Interruption of database-connections tomorrow [19:54:43] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3618 [19:55:03] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1257970.000000 [20:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:07:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14715.000000 [20:15:46] DaBPunkt, how does sql-load-balancer work? [20:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:16:06] I thought that converting "sql-s2-user" to the real server was done by dns... [20:18:02] Platonides: it's a software callled ha-apache. It listen at an ip-adress (for example 10.24.1.102) and than connects to the real server. It also makes checks if a server is reachable [20:21:26] We used a DNS-system called wita in the past. At some point River changed that [20:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:24:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:24:41] but what's the benefit? :S [20:24:58] so that if the server goes down another machine can take that role? [20:30:27] Platonides: A differnce is that the DNS-system checked only if the ip was reachable. The ha-pache also checks that mysql is running. [20:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:38:42] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3496 [20:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7088 [20:41:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103618 MB (17% inode=99%): [20:45:52] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2140.000000 [20:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19168 [20:50:53] / on ortelius is WARNING: DISK WARNING - free space: / 6232 MB (20% inode=92%): [20:52:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4667.000000 [20:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19254.000000 [20:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:48] DaBPunkt: JIRA says I don't exist. do you know about that or I should talk to nosy? [20:57:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1259487.000000 [20:58:03] this has been going on for many months. for some reason i thought maybe it had been fixed [20:58:42] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3642 [21:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:00:46] who has root privileges on mysql? [21:01:23] the roots [21:01:48] talk with DaBPunkt / Nosy [21:02:05] Platonides: thanks [21:02:09] reba: ping [21:02:34] (you can also open a jira bug or email them) [21:03:05] Jeremy Baron * Re: [Toolserver-l] Jira Auth broken [21:03:13] Platonides: The bug is already opened (in reality two) and there is the solution, is only a set of commands to give [21:03:56] hi reba! [21:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17397.000000 [21:10:29] Krinkle: do you exist? [21:13:12] jeremyb_: ? [21:13:37] 19 20:57:47 < jeremyb_> DaBPunkt: JIRA says I don't exist. do you know about that or I should talk to nosy? [21:13:41] 19 20:58:02 < jeremyb_> this has been going on for many months. for some reason i thought maybe it had been fixed [21:13:43] jeremyb_: reba polls a Gmail account every minute. It can take a bit. [21:13:44] Krinkle: and see on list [21:13:55] Susan: it responded! [21:14:07] And how. [21:15:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [21:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:18:02] jeremyb_: I can log into Jira just fine. [21:18:18] Krinkle: i don't exist :( [21:18:28] That's.. complicated. [21:18:39] I see dead people. [21:18:51] {{fact}} [21:19:23] jeremyb_, try to impove your thinking skills [21:19:33] Krinkle: Do you use your Toolserver credentials to log in? [21:19:39] *improve [21:19:46] scfc_de: Define "Toolserver credentials" [21:19:54] scfc_de: There's LDAP, SSH and SSO. [21:20:00] scfc_de: JIRA uses SSO. [21:20:06] (Single-Sign-On) [21:20:18] used for the wiki and the atlantian products such as Jira. [21:20:52] Krinkle: Sorry, messed up. [21:21:03] jeremyb_: Which credentials do you use? :-) [21:21:06] [[LDAP]] 10https://wiki.toolserver.org/w/index.php?diff=7863&oldid=7454&rcid=21678 * Krinkle * (+8) () [21:21:18] https://wiki.toolserver.org/view/User_authentication [21:21:43] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:21:53] / on ortelius is OK: DISK OK - free space: / 6343 MB (21% inode=92%): [21:22:06] scfc_de: none. i'm just trying to do a password reset [21:23:10] oh, you know what... i should try on the wiki [21:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:24:07] except that the wiki says to reset via JIRA [21:28:55] proof that i did once exist: https://jira.toolserver.org/browse/ACC-27 [21:29:06] and https://wiki.toolserver.org/w/index.php?title=Special:Log&user=Jeremyb [21:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:31:32] jeremyb_: Hmmm. Dumb question: Your email address is still the same? [21:31:44] scfc_de: yes [21:32:05] scfc_de: see the mailing list [21:32:34] scfc_de: http://lists.wikimedia.org/pipermail/toolserver-l/2013-March/005814.html [21:33:37] If I choose "Can't login", "Password forgotten" on JIRA, and enter my login name "Tim.Landscheidt" which works perfectly fine for login, I also get the message "No user with that username exists." [21:35:10] scfc_de: and if you try to look up your username? [21:36:06] jeremyb_: Where? [21:36:12] scfc_de: on the same form [21:36:16] radio button [21:36:33] jeremyb_: You mean look up user name by email address? [21:36:51] yes [21:37:01] I get "Your username has been sent to you via email." [21:37:45] jeremyb_: But when I enter your username on *wiki* and request new password, I get the message "Error sending mail: There is no e-mail address recorded for user 'Jeremyb'." [21:38:07] right but the wiki says to do resets at jira [21:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:38:15] Susan: Are you sysop on wiki? [21:38:29] when i go to https://jira.toolserver.org/secure/IssueNavigator.jspa?mode=show => issue attributes => reporter=specify user, click the people icon => search for full name of tim [21:38:45] that gives me a result of tim's full username [21:39:10] when i search for jeremy or jeremyb in the same place then i get no results [21:39:32] scfc_de [21:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8195 [21:39:42] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3482 [21:39:54] scfc_de: Yep. [21:40:27] Susan: Could you see if the user "Jeremyb" has an email address set? [21:40:42] (Don't know how that interacts with SSO, but what the heck.) [21:40:46] Is that something admins can check? [21:40:49] https://wiki.toolserver.org/view/Special:EmailUser/jeremyb says... [21:40:54] > This user has not specified a valid e-mail address. [21:41:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103353 MB (17% inode=99%): [21:41:14] huh [21:41:16] interesting [21:41:43] Susan: I don't know. Has the interface some way to set an email address? [21:41:52] For another user? Nope. :-) [21:42:03] I can see an account exists for "jeremyb", though. [21:42:13] And according to MediaWiki, it has no authenticated e-mail address. [21:42:36] Well, could that be a pattern?! [21:42:41] Looks like MediaWiki e-mail is broken, though. [21:42:47] https://wiki.toolserver.org/view/Special:EmailUser/MZMcBride doesn't work. [21:43:02] can you normally email yourself? [21:43:07] 1.18alpha, heh. [21:43:13] I tried /Dab as well. [21:43:28] https://wiki.toolserver.org/view/Special:ListGroupRights [21:43:33] emailconfirmed? [21:43:35] NFI, sorry. [21:44:34] Susan: "NFI"? [21:44:53] no f***king idea. i assume [21:45:13] well would be nice to exist again. sometime before the end of the month maybe? [21:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:46:58] jeremyb_: You probably need to find some root to see if they can add your email address to SSO. Or just create a new account (don't know how much JIRA stuff you've done already). [21:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20710 [21:51:37] Susan: so can you [[special:emailuser]] tim? [21:52:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5757.000000 [21:52:40] I don't think e-mail works for anyone. [21:52:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20711.000000 [21:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:53:42] MySQL slave on z-dat-s2-b is OK: Uptime: 427239 Threads: 29 Questions: 663393115 Slow queries: 6618 Opens: 4083629 Flush tables: 1 Open tables: 256 Queries per second avg: 1552.744 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1707 [21:56:17] hrmmm [21:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1253919.000000 [21:58:35] But with a different error message than for Jeremyb. [22:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:07:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19344.000000 [22:15:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6416.000000 [22:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8458 [22:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103117 MB (16% inode=99%): [22:43:33] scfc_de: Still about? [22:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:48:57] Susan: Yep. [22:49:52] scfc_de: It seems https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Unused_non-free_files has stopped updating. [22:50:02] I'm wondering if you've gotten any mail about that. [22:50:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21131 [22:50:17] I looked at dbreps@hawthorn and I see it's in the crontab. [22:50:34] I tried manually running the qcronsub command just a few minutes ago. [22:52:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21162.000000 [22:53:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6794.000000 [22:53:05] Maciej Jaros * Re: [Toolserver-l] [TS logo] Fwd: Free as in Wikimedia Foundation [22:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:40] Susan: In the past, a lot of queries were killed (even short running ones like this). So it could have been one of those. But I don't know exactly as I usually ignore errors like that because they are (somewhat) transient. I'll put it on my watchlist. [22:54:24] Susan: Did you the qcronsub as dbreps? [22:57:18] Yes. [22:57:29] dbreps@hawthorn:~/var/log$ qcronsub -b y -j y -l arch=lx -l h_rt=0:15:00 -l virtual_free=60M -l sql-s1-rr=1 -o ~/var/log ~/bin/unusednonfree.p [22:57:43] Hmmm. [22:57:44] https://en.wikipedia.org/w/index.php?title=Wikipedia:Database_reports/Unused_non-free_files&action=history [22:57:47] Seems like it ran. [22:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1242730.000000 [22:59:06] scfc_de: Okay, I updated .forward to include my username as well. [23:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:00:04] Tammy Hughes * [Toolserver-l] toolserver [23:00:58] Susan: Okay. [23:02:07] ? [23:02:13] tammy???! [23:02:40] scfc_de: Yeah, the transient errors drive me crazy. [23:06:06] OQ * Re: [Toolserver-l] [TS logo] Fwd: Free as in Wikimedia Foundation [23:06:06] Matthew Bowker * Re: [Toolserver-l] toolserver [23:08:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20422.000000 [23:15:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8815.000000 [23:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:39:32] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7120 [23:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102879 MB (16% inode=99%): [23:46:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:47:44] nacht ts [23:50:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21674 [23:52:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21763.000000 [23:53:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7230.000000 [23:53:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:57:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1231807.000000