[00:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:03:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 584260 MB (10% inode=70%): [00:04:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [00:07:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3113.000000 [00:10:42] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2145 [00:14:32] MySQL slave on z-dat-s7-a is OK: Uptime: 3412313 Threads: 8 Questions: 1680247510 Slow queries: 76787 Opens: 5175143 Flush tables: 1 Open tables: 6765 Queries per second avg: 492.407 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [00:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8330 [00:26:42] aliasd on nightshade is CRITICAL: Connection refused [00:27:42] MySQL slave on z-dat-s6-a is OK: Uptime: 3413097 Threads: 12 Questions: 1717287164 Slow queries: 184160 Opens: 5733306 Flush tables: 1 Open tables: 4281 Queries per second avg: 503.146 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1711 [00:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97974 MB (16% inode=99%): [00:45:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [00:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:47:12] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3548.000000 [00:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:47:22] http://www.mediawiki.org/w/index.php?title=Toolserver/List_of_Tools <-- What have I done wrong? [00:51:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18861 [00:53:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18909.000000 [00:58:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1155440.000000 [01:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:02:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46542 MB (4% inode=99%): [01:03:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 581864 MB (10% inode=70%): [01:13:12] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1681.000000 [01:21:38] nacht ts [01:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9607 [01:26:42] aliasd on nightshade is CRITICAL: Connection refused [01:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:41:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97801 MB (16% inode=99%): [01:44:44] Hey guys, is sql-s4-user down? [01:45:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5394.000000 [01:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:46:53] $ mysql -hsql-s4-user [01:46:53] ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 [01:46:57] Guess so. [01:47:00] Not sure why. [01:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:51:01] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19446 [01:53:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19486.000000 [01:58:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1144265.000000 [02:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:03:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 584313 MB (10% inode=70%): [02:20:54] [[Talk:~holek/cite-gen/index.php]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7865&rcid=21681 * Lzhpdcde * (+4414) (€œBut keep in mind his education) [02:21:43] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:21:52] [[Talk:~holek/cite-gen/index.php]] ! 10https://wiki.toolserver.org/w/index.php?diff=7866&oldid=7865&rcid=21682 * Lzhpdcde * (+4765) (/* €œNo, no! ” cried Pavel Petrovich having */ new section) [02:22:34] [[Talk:~holek/cite-gen/index.php]] ! 10https://wiki.toolserver.org/w/index.php?diff=7867&oldid=7866&rcid=21683 * Lzhpdcde * (+3957) (/* excitement, seemed to be however not */ new section) [02:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:26:41] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10508 [02:26:43] aliasd on nightshade is CRITICAL: Connection refused [02:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97676 MB (16% inode=99%): [02:45:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6229.000000 [02:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:51:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20499 [02:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20477.000000 [02:55:31] /mnt user-store on rosemary is OK: DISK OK - free space: /mnt 586782 MB (11% inode=70%): [02:58:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1134294.000000 [03:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:07:32] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1990 [03:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11559 [03:26:42] aliasd on nightshade is CRITICAL: Connection refused [03:30:12] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2120.000000 [03:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:33:34] [[Special:Log/newusers]] create 10 * Paddleboard * (New user account) [03:36:00] [[User talk:Paddleboard]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7868&rcid=21685 * Paddleboard * (+300) (Created page with "Play surfing with our best products of surfboards and stand up paddle boards with an affordable price. Pop Paddle Boards has been one of the most famous brand names for the produ...") [03:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:40:32] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3605 [03:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97528 MB (16% inode=99%): [03:45:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7150.000000 [03:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:49:50] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02Talk:~holek/cite-gen/index.php10]]": spam) [03:50:02] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Lzhpdcde10]] with an expiry time of infinite (account creation disabled): spam) [03:50:11] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Paddleboard10]]": spam) [03:50:25] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Paddleboard10]] with an expiry time of infinite (account creation disabled): spam) [03:51:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20990 [03:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21123.000000 [03:58:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1120747.000000 [04:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:21:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:24:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12584 [04:26:42] aliasd on nightshade is CRITICAL: Connection refused [04:30:12] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1867.000000 [04:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:32:12] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1784.000000 [04:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:40:32] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5764 [04:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97397 MB (16% inode=99%): [04:45:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7683.000000 [04:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:51:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22021 [04:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:54:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22028.000000 [04:59:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1099703.000000 [05:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14122 [05:26:42] aliasd on nightshade is CRITICAL: Connection refused [05:31:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:40:32] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8555 [05:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97262 MB (15% inode=99%): [05:45:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9867.000000 [05:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:51:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21645 [05:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21703.000000 [05:59:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1088008.000000 [06:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:24:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14931 [06:26:42] aliasd on nightshade is CRITICAL: Connection refused [06:30:08] DeltaQuad Wikipedia * Re: [Toolserver-l] Interruption of database-connections tomorrow [06:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:40:32] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11561 [06:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97127 MB (15% inode=99%): [06:45:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11695.000000 [06:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:51:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21252 [06:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:53:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21331.000000 [06:59:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1084305.000000 [07:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16367 [07:26:42] aliasd on nightshade is CRITICAL: Connection refused [07:28:02] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2186.000000 [07:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:40:32] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14805 [07:41:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96955 MB (15% inode=99%): [07:45:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13796.000000 [07:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:51:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23039 [07:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:53:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:54:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [07:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22998.000000 [07:59:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1079829.000000 [08:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:10:42] aliasd on nightshade is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [08:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:26:41] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16910 [08:31:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:31:33] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:31:41] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:40:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:40:32] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15426 [08:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96736 MB (15% inode=99%): [08:45:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13497.000000 [08:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:47:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:51:02] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3532.000000 [08:51:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23351 [08:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:53:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23448.000000 [08:58:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 585068 MB (10% inode=70%): [09:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1077047.000000 [09:00:02] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1779.000000 [09:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:25:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16362 [09:31:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:31:33] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:36:52] Free Memory on damiana is WARNING: WARNING - 6.2% (518800 kB) free! [09:40:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:40:32] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6000 [09:41:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96515 MB (15% inode=99%): [09:45:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9373.000000 [09:46:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:47:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:48:52] Free Memory on damiana is OK: OK - 7.1% (594692 kB) free. [09:49:32] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3358 [09:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24192 [09:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:53:21] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:32] MySQL slave on z-dat-s7-a is OK: Uptime: 3447113 Threads: 4 Questions: 1709199333 Slow queries: 77630 Opens: 5314817 Flush tables: 1 Open tables: 6814 Queries per second avg: 495.835 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1770 [09:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24188.000000 [09:55:52] Free Memory on damiana is WARNING: WARNING - 5.2% (434264 kB) free! [09:58:31] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 584819 MB (10% inode=70%): [10:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1075069.000000 [10:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:15:51] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (352700 kB) free! [10:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:26:12] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [10:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17522 [10:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:40:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:40:52] Free Memory on damiana is OK: OK - 13.2% (1108496 kB) free. [10:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96274 MB (15% inode=99%): [10:45:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4286.000000 [10:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:51:02] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3506.000000 [10:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24126 [10:53:13] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:53:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24269.000000 [10:55:52] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2169.000000 [10:58:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 582626 MB (10% inode=70%): [11:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1071233.000000 [11:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:12:03] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1755.000000 [11:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:19:52] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [11:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:26:12] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [11:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18797 [11:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:40:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96054 MB (15% inode=99%): [11:47:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:51:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25484 [11:53:13] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:53:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25344.000000 [11:58:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 579998 MB (10% inode=70%): [12:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1066939.000000 [12:00:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:10:52] Free Memory on damiana is WARNING: WARNING - 6.2% (518004 kB) free! [12:14:53] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3094.000000 [12:15:13] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2771 [12:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:16:52] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1047.000000 [12:17:13] MySQL slave on cassia is OK: Uptime: 539 Threads: 8 Questions: 249055 Slow queries: 17 Opens: 94 Flush tables: 1 Open tables: 87 Queries per second avg: 462.68 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 744 [12:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:26:41] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19208 [12:31:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:31:31] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:31:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420340 kB) free! [12:38:52] Free Memory on damiana is WARNING: WARNING - 5.1% (429524 kB) free! [12:38:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2208.000000 [12:39:02] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [12:39:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (423132 kB) free! [12:40:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95812 MB (15% inode=99%): [12:47:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:49:08] DaB. * Re: [Toolserver-l] Interruption of database-connections tomorrow [12:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25239 [12:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:53:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25271.000000 [12:58:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 578138 MB (10% inode=70%): [13:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1061148.000000 [13:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:02:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46351 MB (4% inode=99%): [13:02:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3648.000000 [13:03:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3654.000000 [13:12:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (407336 kB) free! [13:13:52] Free Memory on damiana is WARNING: WARNING - 5.2% (438836 kB) free! [13:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18718 [13:26:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421052 kB) free! [13:27:52] Free Memory on damiana is WARNING: WARNING - 5.1% (426624 kB) free! [13:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:40:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:40:21] hello all [13:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95579 MB (15% inode=99%): [13:47:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24756 [13:52:52] Free Memory on damiana is WARNING: WARNING - 7.0% (586984 kB) free! [13:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:53:21] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:53] Free Memory on damiana is OK: OK - 7.2% (600716 kB) free. [13:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24678.000000 [13:58:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 576088 MB (10% inode=70%): [14:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1052573.000000 [14:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:02:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (366248 kB) free! [14:02:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7248.000000 [14:03:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7254.000000 [14:05:22] [[Special:Log/newusers]] create 10 * Pratyya Ghosh * (New user account) [14:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:25:32] / on willow is WARNING: DISK WARNING - free space: / 14770 MB (14% inode=98%): [14:25:33] /tmp on willow is WARNING: DISK WARNING - free space: / 14770 MB (14% inode=98%): [14:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18101 [14:28:32] / on willow is CRITICAL: DISK CRITICAL - free space: / 11079 MB (10% inode=97%): [14:28:32] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11079 MB (10% inode=97%): [14:29:52] Free Memory on damiana is WARNING: WARNING - 5.1% (427140 kB) free! [14:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:31:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (406964 kB) free! [14:32:42] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:32:52] Free Memory on damiana is WARNING: WARNING - 6.5% (542788 kB) free! [14:34:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:41:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95337 MB (15% inode=99%): [14:46:32] / on willow is OK: DISK OK - free space: / 26684 MB (25% inode=99%): [14:46:32] /tmp on willow is OK: DISK OK - free space: / 26684 MB (25% inode=99%): [14:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:51:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24068 [14:52:33] fale: Re http://www.mediawiki.org/wiki/Toolserver/List_of_Tools, you were missing "]"s :-). [14:53:13] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:53:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23975.000000 [14:57:32] /mnt user-store on rosemary is OK: DISK OK - free space: /mnt 588108 MB (11% inode=70%): [15:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1043560.000000 [15:00:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:02:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10849.000000 [15:03:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10855.000000 [15:16:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:20:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (399420 kB) free! [15:21:53] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:24:52] Free Memory on damiana is OK: OK - 8.0% (673444 kB) free. [15:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18428 [15:31:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:31:40] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:35:52] Free Memory on damiana is CRITICAL: CRITICAL - 3.7% (313188 kB) free! [15:36:52] Free Memory on damiana is WARNING: WARNING - 5.1% (426220 kB) free! [15:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95067 MB (15% inode=99%): [15:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [15:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22857 [15:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:53:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22905.000000 [16:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1037909.000000 [16:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:02:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11451.000000 [16:03:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11637.000000 [16:05:52] Free Memory on damiana is CRITICAL: CRITICAL - 3.2% (271040 kB) free! [16:13:52] Free Memory on damiana is WARNING: WARNING - 5.3% (445988 kB) free! [16:16:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:24:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3422.000000 [16:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:25:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (408008 kB) free! [16:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18820 [16:29:53] Free Memory on damiana is WARNING: WARNING - 5.3% (444868 kB) free! [16:30:52] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1595.000000 [16:31:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:31:33] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:35:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94832 MB (15% inode=99%): [16:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22380 [16:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:53:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22366.000000 [17:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1031248.000000 [17:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:03:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4079.000000 [17:06:02] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3428.000000 [17:13:02] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1714.000000 [17:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17803 [17:31:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94553 MB (15% inode=99%): [17:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21070 [17:53:13] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:53:21] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21054.000000 [18:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1030853.000000 [18:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:01:53] Free Memory on damiana is WARNING: WARNING - 5.7% (474076 kB) free! [18:02:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (381392 kB) free! [18:21:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16732 [18:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:31:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:41:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94328 MB (15% inode=99%): [18:42:53] Free Memory on damiana is WARNING: WARNING - 6.2% (522308 kB) free! [18:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:48:52] Free Memory on damiana is OK: OK - 7.1% (591276 kB) free. [18:51:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20328 [18:53:13] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:53:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20346.000000 [19:00:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1029166.000000 [19:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:13:52] Free Memory on damiana is WARNING: WARNING - 5.9% (494804 kB) free! [19:20:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 584723 MB (10% inode=70%): [19:21:53] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:23:05] hmm something's off [19:23:08] * DarkoNeko checks the ML [19:24:52] Free Memory on damiana is OK: OK - 7.1% (594980 kB) free. [19:25:00] I had a cut from 13h to ~19h (french time UTC+1) in my cronsubs [19:25:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:26:37] urg, damnit. it looks like someone is filling the pcrocess list again [19:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17312 [19:30:00] "shuaib" seems to have 102 processes running, mainly "python /home/shuaib/pywikipedia/checkimages.py" [19:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:31:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (402496 kB) free! [19:32:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:40:46] So, without SGE :/ ? [19:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94039 MB (15% inode=99%): [19:41:49] Maybe DaB. or nosy can help? [19:42:17] what is the problem? [19:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20362 [19:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:53:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20367.000000 [20:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1025650.000000 [20:01:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:11:14] DarkoNeko: which server? [20:15:42] I was on willow [20:17:47] I guess I have to out-comment some more cronlines than… [20:18:28] thanks :) I have a pretty hard time with Sun's "top" so there might be another I missed. [20:20:32] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 581933 MB (10% inode=70%): [20:23:25] user "amir" seems to have a lot of processes too (68) trying to check if they're all same script on the same wiki [20:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:26:41] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18010 [20:27:14] unrelated : whenever I try to pipe commandas (ie ps aux | grep something) it seems truncated to 80 chars per line ; is there something I can do about that ? [20:29:12] DaBPunkt: Add a few w's to your command line. :-0 [20:29:20] ps auxwww [20:29:58] * DaBPunkt redirects that to DarkoNeko [20:30:00] eeh ? ^^; [20:30:06] Just ww [20:30:30] Yes, that was for DarkoNeko obviously. I'm pretty sure DaBPunkt knew that already. [20:30:38] ooh, thank you [20:30:51] Coren: I have to admit that I didn't knew [20:31:24] DaBPunkt: Some of those old tools have very obscure options. :-) [20:31:32] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:31:41] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:31:42] oops [20:32:13] I didn't know about the "ww" option. [20:32:14] Coren: ps is always strange (see the xkcd-comic short time ago) [20:32:30] I usually just make my terminal window larger, heh. [20:32:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:33:47] Susan: that's not working under Solaris [20:33:52] mine is fairly large, hence my frustration with the truncation [20:34:20] Solaris is evil. [20:34:35] ...yes [20:34:40] yes [20:35:09] well honestly, it's just that it throws me off some bit compared to what I do on linux, subtle annoying differences on a lot of places [20:36:17] Right. [20:36:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:37:52] Free Memory on damiana is WARNING: WARNING - 5.1% (430520 kB) free! [20:38:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (399832 kB) free! [20:39:55] from warning to critical in 1min… [20:40:42] uwah ? [20:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93744 MB (15% inode=99%): [20:43:55] DarkoNeko: pardon? [20:45:14] surprise onomatopea [20:45:19] what went critical ? [20:45:39] memory usage on the active ha-node [20:45:50] oh [20:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:51:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20921 [20:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:53:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:52] Free Memory on damiana is WARNING: WARNING - 5.8% (485864 kB) free! [20:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20941.000000 [21:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1017200.000000 [21:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:08:52] Free Memory on damiana is OK: OK - 7.2% (601484 kB) free. [21:15:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (350776 kB) free! [21:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 579181 MB (10% inode=70%): [21:23:52] Free Memory on damiana is WARNING: WARNING - 5.6% (465496 kB) free! [21:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18854 [21:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:31:43] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:41:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93420 MB (15% inode=99%): [21:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:47:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:51:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21378 [21:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:53:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:54:52] Free Memory on damiana is CRITICAL: CRITICAL - 3.9% (325160 kB) free! [21:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21435.000000 [21:59:52] Free Memory on damiana is WARNING: WARNING - 6.5% (545740 kB) free! [22:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 993682.000000 [22:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:03:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (396572 kB) free! [22:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 576439 MB (10% inode=70%): [22:24:52] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:26:43] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20105 [22:29:52] Free Memory on damiana is WARNING: WARNING - 5.1% (426008 kB) free! [22:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:31:42] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:33:52] Free Memory on damiana is OK: OK - 7.2% (607500 kB) free. [22:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93112 MB (15% inode=99%): [22:46:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:50:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (365596 kB) free! [22:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21278 [22:53:12] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:53:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21313.000000 [23:00:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 978241.000000 [23:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:06:52] Free Memory on damiana is WARNING: WARNING - 5.1% (424428 kB) free! [23:20:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 572876 MB (10% inode=69%): [23:24:53] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:26:42] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19663 [23:31:42] APT on z-dat-s2-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:31:43] APT on z-dat-s1-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:32:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:34:52] Free Memory on damiana is OK: OK - 7.3% (612944 kB) free. [23:37:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:41:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92834 MB (15% inode=99%): [23:47:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:48:53] Free Memory on damiana is WARNING: WARNING - 6.7% (563604 kB) free! [23:51:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21514 [23:53:22] APT on z-dat-s5-b is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:53:31] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21488.000000 [23:59:37] nacht ts [23:59:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (373308 kB) free!