[00:00:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:00:35] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [00:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87952.000000 [00:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:16:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88253 [00:21:44] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2014.000000 [00:31:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 495422 MB (9% inode=65%): [00:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39444.000000 [00:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:41:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:42:04] aliasd on nightshade is CRITICAL: Connection refused [00:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16600.000000 [00:54:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72826 MB (11% inode=99%): [00:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [00:57:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:59:33] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [00:59:54] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:00:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:00:34] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [01:06:28] [[Special:Log/newusers]] create 10 * Liuxinyu970226 * (New user account) [01:08:04] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 33228 MB (3% inode=99%): [01:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88984.000000 [01:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:16:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88760 [01:21:43] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2700.000000 [01:31:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 495377 MB (9% inode=65%): [01:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41231.000000 [01:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:41:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:42:04] aliasd on nightshade is CRITICAL: Connection refused [01:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19482.000000 [01:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70779 MB (11% inode=99%): [01:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [01:57:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:59:34] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [01:59:54] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:00:34] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [02:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89176.000000 [02:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89383 [02:21:44] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2622.000000 [02:31:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 495017 MB (9% inode=65%): [02:32:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42871.000000 [02:33:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:41:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:42:04] aliasd on nightshade is CRITICAL: Connection refused [02:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:48:04] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2019.000000 [02:48:43] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2047 [02:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20760.000000 [02:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72610 MB (11% inode=99%): [02:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [02:57:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:59:34] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [02:59:54] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:00:34] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [03:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88844.000000 [03:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88286 [03:21:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [03:21:44] MySQL slave on z-dat-s5-b is OK: Uptime: 103119 Threads: 7 Questions: 191126087 Slow queries: 1076 Opens: 71217 Flush tables: 1 Open tables: 256 Queries per second avg: 1853.451 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1786 [03:22:05] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1734.000000 [03:31:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 494274 MB (9% inode=65%): [03:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44226.000000 [03:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:35:44] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1965 [03:36:04] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1968.000000 [03:41:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:42:05] aliasd on nightshade is CRITICAL: Connection refused [03:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20398.000000 [03:51:44] MySQL slave on z-dat-s5-b is OK: Uptime: 104919 Threads: 6 Questions: 191850093 Slow queries: 1105 Opens: 116736 Flush tables: 1 Open tables: 256 Queries per second avg: 1828.554 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1670 [03:52:04] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1626.000000 [03:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72504 MB (11% inode=99%): [03:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [03:57:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:59:34] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [03:59:54] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:01:34] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [04:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88553.000000 [04:12:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88846 [04:21:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4200.000000 [04:31:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493723 MB (9% inode=65%): [04:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46023.000000 [04:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:41:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:42:04] aliasd on nightshade is CRITICAL: Connection refused [04:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21548.000000 [04:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72395 MB (11% inode=99%): [04:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [04:57:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:59:44] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [05:00:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:01:43] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [05:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90711.000000 [05:08:44] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3578.000000 [05:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90745 [05:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:32:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 494623 MB (9% inode=65%): [05:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47919.000000 [05:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:41:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:42:04] aliasd on nightshade is CRITICAL: Connection refused [05:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21568.000000 [05:53:34] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [05:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72269 MB (11% inode=99%): [05:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [05:57:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:59:44] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:01:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:01:45] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:03:44] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1792.000000 [06:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90839.000000 [06:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90104 [06:22:04] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:31:14] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [06:32:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 494443 MB (9% inode=65%): [06:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49626.000000 [06:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:41:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:42:04] aliasd on nightshade is CRITICAL: Connection refused [06:46:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20702.000000 [06:53:34] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [06:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72119 MB (11% inode=99%): [06:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:58:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:59:44] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [07:01:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:01:44] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [07:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88419.000000 [07:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:12:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88601 [07:23:04] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [07:31:13] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [07:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:31:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [07:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51035.000000 [07:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 494219 MB (9% inode=65%): [07:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:41:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:43:05] aliasd on nightshade is CRITICAL: Connection refused [07:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20315.000000 [07:53:34] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [07:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71981 MB (11% inode=99%): [07:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [07:58:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:59:44] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:01:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:01:44] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86899.000000 [08:12:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86394 [08:23:04] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:31:14] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:31:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [08:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52261.000000 [08:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 494030 MB (9% inode=65%): [08:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:41:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:43:04] aliasd on nightshade is CRITICAL: Connection refused [08:46:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23212.000000 [08:53:34] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [08:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71870 MB (11% inode=99%): [08:55:05] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:58:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:59:43] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:01:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:02:43] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:09:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87490.000000 [09:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87344 [09:23:04] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:31:14] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [09:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53907.000000 [09:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493653 MB (9% inode=65%): [09:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:41:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:43:04] aliasd on nightshade is CRITICAL: Connection refused [09:45:33] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 47646 MB (9% inode=99%): [09:46:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24631.000000 [09:53:34] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [09:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71780 MB (11% inode=99%): [09:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:58:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:59:44] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:01:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [10:02:44] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:09:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87536.000000 [10:12:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:12:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88079 [10:23:04] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:31:13] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [10:31:54] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2191 [10:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56100.000000 [10:33:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493450 MB (9% inode=65%): [10:33:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:34:43] Load avg. on z-dat-s2-b is WARNING: WARNING - load average: 18.63, 16.25, 13.17 [10:38:25] do we have a tool laying around that does a sort of git blame but for text in a page, so you can see which revision introduced some phrase? [10:41:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:43:04] aliasd on nightshade is CRITICAL: Connection refused [10:50:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28006.000000 [10:50:30] nm found this: http://wikipedia.ramselehof.de/wikiblame.php?lang=en&article=Wikipedia:WikiBlame [10:53:33] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [10:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71698 MB (11% inode=99%): [10:55:07] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:55:31] which unfortunately returns no results for text known to be in a page (and not in a template parameter) [10:55:34] grrrr [10:56:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3644 [10:58:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:59:44] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:01:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:02:44] Load avg. on z-dat-s2-b is OK: OK - load average: 11.05, 13.68, 14.77 [11:02:44] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:09:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88453.000000 [11:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:16:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88450 [11:23:04] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:31:14] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [11:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58130.000000 [11:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493184 MB (9% inode=65%): [11:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:42:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:43:05] aliasd on nightshade is CRITICAL: Connection refused [11:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31589.000000 [11:50:44] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2193.000000 [11:51:04] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2203.000000 [11:53:34] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [11:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71577 MB (11% inode=99%): [11:55:04] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:56:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6946 [11:58:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:59:44] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:01:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:02:44] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:07:44] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.41, 25.29, 15.44 [12:09:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88342.000000 [12:09:44] Load avg. on ortelius is WARNING: WARNING - load average: 15.46, 23.57, 16.17 [12:11:44] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2168.000000 [12:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:12:44] Load avg. on ortelius is OK: OK - load average: 3.34, 14.30, 13.79 [12:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3632.000000 [12:15:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3643.000000 [12:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88720 [12:23:04] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:28:44] s4 replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [12:31:14] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [12:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61692.000000 [12:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 492837 MB (9% inode=65%): [12:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:42:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:43:04] aliasd on nightshade is CRITICAL: Connection refused [12:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35189.000000 [12:50:20] [[صفحهٔ اصلی]] ! 10https://wiki.toolserver.org/w/index.php?diff=7891&oldid=5958&rcid=21725 * Javadyou * (+15) () [12:53:34] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [12:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71457 MB (11% inode=99%): [12:55:14] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:56:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6724 [12:58:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:59:43] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:02:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:44] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:08:04] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31069 MB (3% inode=99%): [13:10:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89630.000000 [13:11:04] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2191.000000 [13:11:33] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2218.000000 [13:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:14:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7233.000000 [13:15:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7243.000000 [13:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89366 [13:23:13] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:29:05] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6803.000000 [13:31:14] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:31:43] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [13:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65291.000000 [13:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 492201 MB (9% inode=65%): [13:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3631.000000 [13:35:34] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3658.000000 [13:37:24] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2203.000000 [13:42:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:43:04] aliasd on nightshade is CRITICAL: Connection refused [13:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38789.000000 [13:52:54] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3597 [13:53:33] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates). [13:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71340 MB (11% inode=99%): [13:55:04] hello all [13:55:14] APT on yucca is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:55:39] hi DaBPunkt, I are you coming to the Amsterdam Hackathon? [13:55:59] multichill: no, because the general member meeting of WMDE is at the same time [13:56:24] Oh, right, that was the same weekend. It's a shame.... [13:56:29] yes [13:57:12] I have no idea why there is no calendar for such things so that the differend chapters can better plan… [13:57:58] We already decided on the day somewhere back in December and WMDE was informed [13:58:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:59:44] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [14:00:16] DaBPunkt: I still have some leftover servers. TS interested or not? I think 8 cores, 32G mem, some local disks + FC [14:01:24] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3643.000000 [14:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:02:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:44] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [14:03:21] multichill: Our racks are full. I requested a new one at WMDE but no reply yet. I will send a reminder. but the offer sounds good [14:03:54] MySQL slave on z-dat-s2-b is OK: Uptime: 139249 Threads: 10 Questions: 217395008 Slow queries: 2629 Opens: 1300529 Flush tables: 1 Open tables: 257 Queries per second avg: 1561.196 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1608 [14:05:08] Doing a bit of spring cleanup so a lot of old stuff is being shutdown. It's still decent, we just moved over to the next generation [14:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88051.000000 [14:12:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:12:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10832.000000 [14:15:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10843.000000 [14:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87583 [14:23:14] APT on sage is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [14:28:15] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:29:44] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:29:44] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:30:04] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10463.000000 [14:30:14] APT on sage is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:30:14] APT on yucca is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:30:34] APT on z-dat-s5-b is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:30:44] APT on z-dat-s2-b is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:31:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68840.000000 [14:33:05] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 491939 MB (9% inode=65%): [14:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6941.000000 [14:35:34] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6884.000000 [14:37:24] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3561.000000 [14:42:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:43:04] aliasd on nightshade is CRITICAL: Connection refused [14:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40994.000000 [14:52:24] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1587.000000 [14:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71212 MB (11% inode=99%): [14:58:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:01:04] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3497.000000 [15:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:02:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:04] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1512.000000 [15:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86798.000000 [15:12:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:12:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11204.000000 [15:15:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11193.000000 [15:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86247 [15:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69596.000000 [15:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 491645 MB (9% inode=65%): [15:33:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:35:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5933.000000 [15:35:34] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5920.000000 [15:42:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:43:04] aliasd on nightshade is CRITICAL: Connection refused [15:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41977.000000 [15:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71077 MB (11% inode=99%): [15:58:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:02:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85794.000000 [16:12:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10214.000000 [16:15:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10209.000000 [16:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85869 [16:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:54] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1942 [16:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70279.000000 [16:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 491397 MB (9% inode=65%): [16:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:33:53] MySQL slave on z-dat-s2-b is OK: Uptime: 148249 Threads: 14 Questions: 225148895 Slow queries: 2798 Opens: 1374698 Flush tables: 1 Open tables: 257 Queries per second avg: 1518.721 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1742 [16:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5609.000000 [16:35:34] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5035.000000 [16:42:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:43:04] aliasd on nightshade is CRITICAL: Connection refused [16:46:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43920.000000 [16:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70948 MB (11% inode=99%): [16:58:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:01:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:02:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:54] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2080 [17:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88783.000000 [17:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9320.000000 [17:15:09] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9317.000000 [17:20:37] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89290 [17:29:54] MySQL slave on z-dat-s2-b is OK: Uptime: 151609 Threads: 12 Questions: 230548048 Slow queries: 2870 Opens: 1404352 Flush tables: 1 Open tables: 256 Queries per second avg: 1520.675 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1796 [17:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72314.000000 [17:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 491080 MB (9% inode=65%): [17:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6390.000000 [17:35:34] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4425.000000 [17:42:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:43:04] aliasd on nightshade is CRITICAL: Connection refused [17:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44622.000000 [17:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70835 MB (11% inode=99%): [17:59:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:02:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91909.000000 [18:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8455.000000 [18:15:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8461.000000 [18:16:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92388 [18:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75111.000000 [18:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 490781 MB (9% inode=65%): [18:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8531.000000 [18:35:33] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4145.000000 [18:42:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:43:04] aliasd on nightshade is CRITICAL: Connection refused [18:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45545.000000 [18:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70721 MB (11% inode=99%): [18:59:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:01:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:02:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94452.000000 [19:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8142.000000 [19:15:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8342.000000 [19:16:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 94210 [19:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75978.000000 [19:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 490378 MB (9% inode=65%): [19:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:33:34] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3587.000000 [19:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8971.000000 [19:43:04] aliasd on nightshade is CRITICAL: Connection refused [19:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44133.000000 [19:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70603 MB (11% inode=99%): [19:54:44] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 28.45, 37.23, 22.96 [19:57:43] Load avg. on ortelius is WARNING: WARNING - load average: 7.20, 23.14, 19.86 [19:59:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:59:44] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 21.13, 23.94, 20.58 [20:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:02:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:51] hello [20:02:55] maintenance tonight again [20:03:02] will again reboot nightshade now [20:08:04] aliasd on nightshade is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [20:11:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94677.000000 [20:12:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:14:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7841.000000 [20:15:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8343.000000 [20:15:08] ok nightshade is online again - seems no trouble this time [20:16:01] ill continue with mayapple [20:16:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:16:06] hmm [20:16:30] nosy, [20:16:48] ah dns is not updated...hang on [20:16:50] phe... [20:16:56] yeah :) [20:17:11] nosy or someone with access, I think this update is needed >> update toolserver.wiki set domain = 'www.wikidata.org' where dbname='wikidatawiki_p'; << [20:17:54] phe: new ip is 185.15.59.201 [20:18:05] They have set the www as canonical [20:18:37] ty nosy [20:18:42] jem-: what? sorry my head is programmed for renumbering currently [20:19:14] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:14] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:25] nosy: Don't worry :) It's an update in the toolserver database [20:19:33] Sensors on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:33] / on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:33] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:33] /tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:34] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:43] NTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:53] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:54] Environment IPMI on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:20:13] I have provided the SQL command needed, to make it easy [20:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 95023 [20:24:33] rebooting mayappe [20:25:31] yarrow is next [20:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75619.000000 [20:32:23] reboot yarrow [20:33:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 489912 MB (9% inode=65%): [20:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:33:34] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2799.000000 [20:34:59] If I "sql toolserver" on nightshade, I get "ERROR 1045 (28000): Access denied for user 'timl'@'185.15.59.201' (using password: YES)". Do the permissions need to be updated, or will this work when DNS is updated? [20:35:03] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8394.000000 [20:37:11] jem-: i just setted the value for wikidata [20:37:20] DaBPunkt: ^^ [20:37:46] scfc_de: nope seems a mysql error... [20:37:52] hrhrhr...ill fix [20:37:54] scfc_de: no. We use a hostname-based auth [20:38:36] but nightshade's DNS-entry was not updated AFAIS [20:38:39] nosy ? [20:38:57] DaBPunkt: i did [20:39:10] probably caching? [20:40:23] Well, what's the TTL then? :-) [20:40:43] new ip propagated down to my house now [20:41:49] I'm restarting the recursors [20:42:04] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:42:04] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:42:04] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:42:05] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:42:05] SMTP on nightshade is OK: SMTP OK - 0.078 sec. response time [20:42:05] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:42:10] worked [20:42:24] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:42:24] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:42:34] NTP on nightshade is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.000301 secs [20:42:44] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:43:00] I'm still getting: "ERROR 1045 (28000): Access denied for user 'timl'@'185.15.59.201' (using password: YES)". [20:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:43:34] NTP on nightshade is OK: NTP OK: Offset -0.000267 secs [20:44:01] scfc_de: which server do you try? [20:45:15] DaBPunkt: On nightshade, "sql toolserver". [20:45:34] It works on yarrow. [20:45:48] And on mayapple. [20:46:07] And still fails on nightshade. [20:47:30] nosy@nightshade:~$ host nightshade [20:47:30] nightshade.toolserver.org has address 91.198.174.201 [20:47:30] nightshade.toolserver.org has IPv6 address 2a02:ec80:101::2:3 [20:47:30] nosy@nightshade:~$ host yarrow [20:47:30] yarrow.toolserver.org has address 185.15.59.216 [20:47:30] yarrow.toolserver.org has IPv6 address 2a02:ec80:101::2:8 [20:47:33] strange... [20:47:44] Sun Grid Engine execd on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:47:44] APT on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:13] APT on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:14] SRaid on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:15] Load avg. on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:15] NTP on yarrow is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset 0.074876 secs [20:48:28] nosy: That's the information from the DNS servers, right? [20:48:34] /tmp on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:35] / on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:35] Sensors on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:36] looks like nightshade can not reach 10.24.1.18 and uses the fall-back [20:48:43] Environment IPMI on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:44] aliasd on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:48:53] which is strange [20:49:04] /tmp on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:49:04] / on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:49:04] SRaid on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:49:04] Sensors on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:49:04] Load avg. on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:49:04] aliasd on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:49:04] Sun Grid Engine execd on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:49:05] Environment IPMI on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:50:07] DaBPunkt: Which fall-back? [20:50:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45358.000000 [20:50:14] google [20:50:24] root@turnera:/etc/opt/ts/namedb/master# host nightshade [20:50:24] nightshade.toolserver.org has address 185.15.59.201 [20:50:24] nightshade.toolserver.org has IPv6 address 2a02:ec80:101::2:3 [20:50:30] DaBPunkt: yes seems so [20:50:56] nosy: did you changed the allow-recusion at the bind-recoursor? [20:51:08] ah I see. no [20:51:13] so the dns master at least knows the correct address [20:51:15] that's the problem [20:51:41] you forgot to changed allow-recursion in the bind-config [20:51:44] I will fix that [20:52:13] NTP on yarrow is OK: NTP OK: Offset 0.096263 secs [20:52:20] Now I can't log into nightshade, and the "host nightshade.toolserver.org" before I logged out gave "Host nightshade.toolserver.org.toolserver.org not found: 5(REFUSED)". [20:53:50] DaBPunkt: forgot firewalling there too [20:53:54] ill fix that [20:54:15] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:28] nosy: ok [20:54:34] Sensors on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:34] / on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:34] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:34] /tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70468 MB (11% inode=99%): [20:54:34] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:39] I will visit the restroom for a moment [20:54:53] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:53] Environment IPMI on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:24] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:47] re [20:59:44] PING on nightshade is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [20:59:44] NTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:59:54] SMTP on ptolemy is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:59:54] PING on hyacinth is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [20:59:56] Logging into nightshade works again, but is very slow. [21:00:04] PING on z-dat-s4-a is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:00:04] PING on willow is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:00:14] SMTP on rosemary is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:14] SMTP on cassia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:14] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:14] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:15] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:19] wait a moment [21:00:24] SMTP on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:24] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:24] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:24] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:24] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:35] PING on nightshade is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms [21:00:35] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:35] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:44] PING on daphne is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:00:44] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:01:34] PING on daphne is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms [21:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:02:04] PING on z-dat-s7-a is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:02:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:54] PING on z-dat-s7-a is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms [21:03:44] SMTP on wolfsbane is OK: SMTP OK - 0.015 sec. response time [21:03:53] PING on z-dat-s4-a is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms [21:04:04] PING on hemlock is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:05:04] PING on willow is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms [21:10:14] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:10:24] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:04] PING on z-dat-s4-a is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95862.000000 [21:11:15] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:54] PING on hemlock is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms [21:12:14] SSH on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:33] NTP on nightshade is OK: NTP OK: Offset 0.000406 secs [21:12:54] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:13:04] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:13:15] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:13:15] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:13:15] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:13:15] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:13:34] SMTP on ptolemy is OK: SMTP OK - 0.224 sec. response time [21:13:51] ok, fixed the recoursor [21:13:53] PING on hyacinth is OK: PING OK - Packet loss = 0%, RTA = 0.32 ms [21:14:03] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.006 second response time [21:14:04] SMTP on adenia is OK: SMTP OK - 0.011 sec. response time [21:14:04] SMTP on z-dat-s5-b is OK: SMTP OK - 0.012 sec. response time [21:14:04] SMTP on ortelius is OK: SMTP OK - 0.008 sec. response time [21:14:04] SMTP on rosemary is OK: SMTP OK - 0.066 sec. response time [21:14:04] SMTP on z-dat-s2-b is OK: SMTP OK - 0.019 sec. response time [21:14:04] SMTP on z-dat-s4-a is OK: SMTP OK - 0.015 sec. response time [21:14:04] SMTP on hemlock is OK: SMTP OK - 0.007 sec. response time [21:14:05] SMTP on z-dat-s3-a is OK: SMTP OK - 0.009 sec. response time [21:14:05] SMTP on z-dat-s6-a is OK: SMTP OK - 0.010 sec. response time [21:14:06] SMTP on cassia is OK: SMTP OK - 0.004 sec. response time [21:14:06] SMTP on z-dat-s7-a is OK: SMTP OK - 0.049 sec. response time [21:14:13] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:13] SMTP on hyacinth is OK: SMTP OK - 0.017 sec. response time [21:14:13] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:13] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:14] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:14] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:14] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:15] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:15] SMTP on sage is OK: SMTP OK - 0.002 sec. response time [21:14:16] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:23] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:24] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:24] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6955.000000 [21:15:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7953.000000 [21:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:18:44] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:44] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:44] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:44] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:44] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:44] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:44] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:45] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:45] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:46] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:46] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:18:47] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:19:44] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:20:04] SRaid on yarrow is OK: OK md0 status=[UU]. [21:20:04] Sensors on yarrow is OK: sensor ok [21:20:04] /tmp on yarrow is OK: DISK OK - free space: /tmp 4093 MB (96% inode=99%): [21:20:04] / on yarrow is OK: DISK OK - free space: / 1591 MB (89% inode=94%): [21:20:04] aliasd on yarrow is OK: TCP OK - 0.001 second response time on port 984 [500 Not found.] [21:20:04] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [21:20:04] Environment IPMI on yarrow is OK: ok: temperature ok fan ok voltage ok chassis ok [21:20:14] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:14] Load avg. on yarrow is OK: OK - load average: 0.08, 0.05, 0.01 [21:20:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:14] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:14] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:14] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:15] nosy: how far is the firewall? [21:20:15] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:15] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:16] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:17] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:20:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:24] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:20:24] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:20:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 96000 [21:20:43] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:43] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:44] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:44] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:44] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:44] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:44] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:44] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:45] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [21:20:58] DaBPunkt: i think i was mistaken there was something missing so i assume i dont need to set anything there [21:21:01] you can look [21:21:06] ipfstat -io [21:21:13] probably you see something [21:21:40] looks good [21:22:20] I'm restarting the mysql-instance on the HA-nodes [21:22:26] DaBPunkt: your dns works now though :) [21:22:34] i just tested [21:23:45] DaBPunkt: just fixed puppet (fw on hemlock) [21:24:00] DaBPunkt: access to toolserver db works now too [21:24:18] still very slow for me [21:25:04] Sensors on nightshade is CRITICAL: Connection refused by host [21:25:04] / on nightshade is CRITICAL: Connection refused by host [21:25:04] /tmp on nightshade is CRITICAL: Connection refused by host [21:25:04] aliasd on nightshade is CRITICAL: Connection refused by host [21:25:04] Sun Grid Engine execd on nightshade is CRITICAL: Connection refused by host [21:26:04] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:26:04] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:26:04] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:26:04] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:26:05] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:30:24] Environment IPMI on nightshade is OK: ok: temperature ok fan ok voltage ok chassis ok [21:30:24] Load avg. on nightshade is OK: OK - load average: 0.07, 0.07, 0.08 [21:30:44] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:30:47] nosy: Thanks, but I'm still seeing "wikidata.org" instead of "www.wikidata.org", is there some lag or replication involved? [21:30:49] DaBPunkt: something is fishy with nightshade itself [21:30:50] nosy: you forget to update nagios' config of nightshade. [21:31:02] all tools are running slow like ls too [21:31:08] yes [21:31:15] puppet should do but i wait [21:31:33] ps aux|grep 1577 [21:31:38] takes ages too [21:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:00] nosy: did you changed the ldap-config and/or nfs? [21:32:07] nope [21:32:19] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74192.000000 [21:32:19] ldap worked [21:32:24] otherwise users could not log in [21:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 489488 MB (9% inode=65%): [21:33:10] DaBPunkt: its no issue on yarrow just nightshade [21:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:33:34] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1925.000000 [21:34:10] nosy: let see if a reboot helps. Set one for 21:40 UTC [21:34:15] DaBPunkt: ok lets see :) [21:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7553.000000 [21:35:19] DaBPunkt: btw - do you know how the ssh known_hosts file was generated? [21:35:57] river used a script AFAIK, but I maintenanced it by hand. [21:37:02] nosy: # Don't edit anything below here, instead run './mk_known_hosts' to re-generate it. [21:37:20] thx [21:37:25] you can find mk_known_hosts in base/files/mk_known_hosts [21:37:31] in puppet [21:37:52] do you have a console open for nightshade? [21:38:17] DaBPunkt: yes but hanging [21:38:28] not hanging [21:38:32] 3 min left [21:38:37] DaBPunkt: [21:39:23] ok [21:39:40] I see you're busy [21:40:34] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1792.000000 [21:40:37] I'll ask again in a while [21:40:38] jem-: where do you see this? [21:40:46] which server? [21:40:57] willow, nosy [21:41:07] The mysql command line client [21:41:11] you do something like sql toolserver there? [21:42:08] I'll copy [21:42:14] paste* [21:42:18] jem@willow:~/wiki$ mysql -se "select domain from toolserver.wiki where dbname='wikidatawiki_p'"; [21:42:22] domain [21:42:24] wikidata.org [21:42:51] really funny [21:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:43:45] jem-: where is the problem with that? [21:43:46] ah there is something fishy with replication probably [21:44:13] DaBPunkt: The canonical url is www.wikidata.org and they said we should avoid wikidata.org [21:44:24] DaBPunkt: i changed the value on damiana [21:44:32] he queries adenia [21:44:41] i probably messed up something? [21:44:42] * jeremyb_ waves nosy [21:44:51] jeremyb_: hello [21:44:55] So they asked me to change the domain that my bot shows up in the [[links]] [21:44:57] nosy: the updater runs only a few times a day [21:44:57] how goes it? [21:45:04] SSH on nightshade is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [21:45:04] SMTP on nightshade is OK: SMTP OK - 0.063 sec. response time [21:45:15] but we can fix that later [21:45:18] jeremyb_: snafu one would say :) [21:45:19] wikidata.org is working [21:45:26] i guess you turned to +0200 very recently [21:45:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 46514 MB (8% inode=99%): [21:45:42] DaBPunkt: but i updated at the wron site? [21:45:48] jeremyb_: In Europe, less than 24 hours ago [21:45:55] nosy: no [21:45:56] jem-: that's the point :) [21:45:58] :) [21:46:04] :) [21:46:06] nightshade is back and it looks better now [21:46:17] scfc_de: can you verify? [21:46:33] DaBPunkt: yes [21:47:03] DaBPunkt: my access there works now as user nosy [21:47:34] Well, if an updater still has to do its job I can wait until tomorrow and review again, no problem [21:47:44] Sun Grid Engine execd on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:48:13] APT on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:48:14] SRaid on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:48:34] /tmp on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:48:34] / on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:48:34] Sensors on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:48:44] aliasd on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:48:44] Environment IPMI on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:48:44] jem-: :) [21:48:44] DaBPunkt: Login is quick and "sql toolserver" works as advertized. Seems the reboot solved whatever what was wrong. [21:49:04] Load avg. on mayapple is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:49:47] scfc_de: ok [21:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44748.000000 [21:51:33] NTP on nightshade is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.010341 secs [21:51:44] Sun Grid Engine execd on mayapple is CRITICAL: Connection refused by host [21:52:04] Load avg. on mayapple is OK: OK - load average: 0.07, 0.04, 0.00 [21:52:13] Restarted nagios on yarrow and mayapple [21:52:14] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:52:14] SRaid on mayapple is OK: OK md0 status=[UUUU]. [21:52:34] / on mayapple is OK: DISK OK - free space: / 1597 MB (89% inode=94%): [21:52:34] /tmp on mayapple is OK: DISK OK - free space: /tmp 4093 MB (96% inode=99%): [21:52:34] Sensors on mayapple is OK: sensor ok [21:52:44] Environment IPMI on mayapple is OK: ok: temperature ok fan ok voltage ok chassis ok [21:52:44] aliasd on mayapple is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [21:54:22] DaBPunkt: damn...i thought it happened automatically [21:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70081 MB (11% inode=99%): [21:55:02] nosy: it should, but the nagios--init-script is broken and the debian people will not update it before next release [21:55:14] oh... [21:58:03] Sun Grid Engine execd on nightshade is CRITICAL: CRITICAL: execd not communicating [21:58:42] DaBPunkt: need to fix mail relaying for the servers too... [22:00:34] NTP on nightshade is OK: NTP OK: Offset -0.024586 secs [22:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:03:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:44] SMTP on turnera is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:45] DaBPunkt: mail works again too [22:06:13] ok ill go now [22:06:17] *wave* [22:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98841.000000 [22:14:14] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:14] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:14] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:14] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:14] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:14] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:14] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:15] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:24] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:24] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:24] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6430.000000 [22:15:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7327.000000 [22:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:16:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:18:43] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:43] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:43] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:43] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:43] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:44] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:44] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:45] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:45] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:46] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:46] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:47] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:47] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:18:48] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:04] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [22:20:13] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:13] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:14] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:14] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:14] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:14] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:20:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 99062 [22:20:44] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:45] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:45] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:45] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:45] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:46] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:46] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:20:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Crypt/DES.pm not found). [22:23:36] How long does it take for a job on the SGE to actually be deleted? [22:28:24] couble of seconds [22:30:42] it's taking minutes, at least [22:30:54] to you too QueenOfFrance? [22:31:30] DaBPunkt: I did the qdel about an hour ago [22:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:12] which job-id? [22:32:19] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73027.000000 [22:32:42] DaBPunkt: 1660416 [22:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 489153 MB (9% inode=65%): [22:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:34:40] I see the porblem. SGE is not active at yarrow because of the maintaince-reboot. Wait a moment [22:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6842.000000 [22:36:03] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [22:36:20] DaBPunkt: btw been meaning to poke you about this for a while, could you take a look at my comments on https://jira.toolserver.org/browse/TS-1589 since you closed it? [22:41:34] QueenOfFrance: I am not able to let database-entries appear. If something is broken there only a complete re-import could fix that [22:43:05] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [22:43:16] DaBPunkt: yeah but it's every single user from wikivoyage that was imported rather than created after wikimedia took over [22:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:43:44] Sun Grid Engine execd on mayapple is OK: Host and Queues Ok [22:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45709.000000 [22:52:54] QueenOfFrance: ok, maybe I can re-import these missing users. [22:53:10] DaBPunkt: <3 [22:53:59] If that's possible, it would be wonderful as it affects all wikivoyages :( [22:54:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69845 MB (11% inode=99%): [23:00:14] FC 0/10 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/10:DOWN:1 UP: OK [23:00:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/21:DOWN:1 UP: OK [23:00:14] FC 0/21 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/21:DOWN:1 UP: OK [23:00:15] FC 0/19 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/19:DOWN:1 UP: OK [23:00:15] eth0 on fsw2-n1-oe16-esams.mgmt is OK: eth0:UP:1 UP: OK [23:00:15] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/2:UP:1 UP: OK [23:00:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/11:UP:1 UP: OK [23:00:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/12:UP:1 UP: OK [23:00:15] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/3:UP:1 UP: OK [23:00:16] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/12:UP:1 UP: OK [23:00:16] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/7:UP:1 UP: OK [23:00:17] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/1:UP:1 UP: OK [23:00:17] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/8:UP:1 UP: OK [23:00:18] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/15:UP:1 UP: OK [23:00:34] SMTP on turnera is OK: SMTP OK - 0.905 sec. response time [23:00:44] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/3:UP:1 UP: OK [23:00:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/16:DOWN:1 UP: OK [23:00:44] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/0:UP:1 UP: OK [23:00:44] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/4:UP:1 UP: OK [23:00:44] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/1:UP:1 UP: OK [23:00:44] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/15:UP:1 UP: OK [23:00:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/5:DOWN:1 UP: OK [23:00:45] FC 0/18 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/18:DOWN:1 UP: OK [23:00:45] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/10:UP:1 UP: OK [23:00:46] FC 0/19 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/19:DOWN:1 UP: OK [23:00:46] FC 0/18 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/18:DOWN:1 UP: OK [23:00:47] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/11:UP:1 UP: OK [23:01:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:01:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:03:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:08] QueenOfFrance: please check if it works for enwikivoyage [23:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98710.000000 [23:12:57] DaBPunkt: it does! [23:13:04] Thanks so much :) [23:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7228.000000 [23:15:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6322.000000 [23:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:16:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:17:39] nacht ts [23:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98372 [23:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71141.000000 [23:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 484796 MB (9% inode=65%): [23:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:35:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5384.000000 [23:43:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47322.000000 [23:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69564 MB (11% inode=99%):