[00:00:04] messed up sorry [00:00:43] Betacommand: should be done within 24h. I would be already done if thyme had enough disc-space and/or I could use the SAN [00:02:26] fale: the job is still running [00:02:47] DaBPunkt: yep, the one that I was looking for was 1567572 and I copied the wrong one -.- [00:03:00] qacct is only for finished jobs [00:11:16] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 3294 MB (10% inode=92%): [00:12:52] I'm new to nagios, but I'm wondering... is "normal" that nagios have these many errors or is because the TS infrastracture is really small for its usage? [00:13:16] / on ortelius is WARNING: DISK WARNING - free space: / 4189 MB (13% inode=92%): [00:14:20] fale: that depens. The "/ on ortelius " is a real warning and I do care about at the moment. The sge-errors on the other hands are just broken plugins [00:15:07] I see :) [00:15:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46441.000000 [00:16:16] Load avg. on nightshade is WARNING: WARNING - load average: 12.73, 15.81, 14.79 [00:16:19] normaly you can ackknowlege a problem in nagios (that makes the warning stops) [00:17:03] but River didn't setup it that way and I have not found time yet to change it [00:18:06] DaBPunkt: I see. Am I wrong or the number of TS root is decreasing over the last years? [00:18:16] Load avg. on nightshade is OK: OK - load average: 11.73, 14.42, 14.40 [00:18:16] / on ortelius is OK: DISK OK - free space: / 10160 MB (33% inode=92%): [00:21:13] fale: I'm not sure. I can't remember a time when the TS had more that 2-3 active roots [00:21:21] that → than [00:21:36] DaBPunkt: so maybe was only my perception [00:25:43] @replag [00:25:44] DaBPunkt: s1-user: 1h 50m 15s [+0.46 s/s]; s2-user-c: error; s2-user-wd: 1d 45m 54s [-0.54 s/s]; s3-user: 3m 18s [-]; s4-user-wd: 12h 32m 55s [-1.66 s/s]; s5-user: 2m 38s [-]; s5-user-c: 2w 3d 2h 19m 47s [+0.61 s/s]; s6-user-wd: 6m 17s [+0.08 s/s] [00:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 117806 MB (2% inode=66%): [00:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:32:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89194.000000 [00:44:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:45:04] DaB. * Re: [Toolserver-l] Error in Revision Table in lbwiki [00:45:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:48:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7381.000000 [00:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7381 [00:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64616 MB (10% inode=99%): [00:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:52:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:54:04] DaB. * Re: [Toolserver-l] No user_properties table on arwiki_p database [00:58:41] Hrmphhhhhh. [00:58:45] Ugly quoting. [00:59:04] MZMcBride * Re: [Toolserver-l] Error in Revision Table in lbwiki [01:02:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51534 MB (5% inode=99%): [01:06:57] nacht ts [01:08:04] is possible to execute "personal" cosmetic changes in pywikipedia? [01:09:06] Its a giant list of categorized regexes, so yes in theory [01:09:33] Dispenser: I guess is not as easy as with user-fixes [01:15:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41854.000000 [01:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 117641 MB (2% inode=66%): [01:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:32:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90748.000000 [01:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:45:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:48:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7020.000000 [01:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7019 [01:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62594 MB (10% inode=99%): [01:51:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:52:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:10:56] Load avg. on ortelius is WARNING: WARNING - load average: 7.95, 20.75, 16.36 [02:14:56] Load avg. on ortelius is OK: OK - load average: 9.22, 14.24, 14.58 [02:15:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37352.000000 [02:26:56] Load avg. on ortelius is WARNING: WARNING - load average: 17.21, 17.63, 16.19 [02:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 116230 MB (2% inode=66%): [02:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:32:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90315.000000 [02:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:45:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:48:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8122.000000 [02:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8131 [02:51:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64376 MB (10% inode=99%): [02:51:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:52:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:59:56] Load avg. on ortelius is OK: OK - load average: 2.43, 9.19, 14.89 [03:15:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33667.000000 [03:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 114599 MB (2% inode=66%): [03:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:32:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89702.000000 [03:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:45:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:48:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9479.000000 [03:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9483 [03:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64205 MB (10% inode=99%): [03:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:53:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:15:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28884.000000 [04:23:36] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1848 [04:24:36] MySQL slave on z-dat-s2-b is OK: Uptime: 105096 Threads: 9 Questions: 170642668 Slow queries: 1461 Opens: 1284693 Flush tables: 1 Open tables: 256 Queries per second avg: 1623.683 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [04:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 113629 MB (2% inode=66%): [04:28:35] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:32:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90258.000000 [04:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:45:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:48:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10839.000000 [04:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10844 [04:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64049 MB (10% inode=99%): [04:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:53:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:15:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25442.000000 [05:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 114687 MB (2% inode=66%): [05:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:32:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86329.000000 [05:34:56] [[Contact]] ! 10https://wiki.toolserver.org/w/index.php?diff=7853&oldid=7553&rcid=21664 * 208.54.83.163 * (-686) (Blanked the page) [05:36:24] [[Contact]] ! 10https://wiki.toolserver.org/w/index.php?diff=7854&oldid=7853&rcid=21665 * 130.126.56.169 * (+686) (Undo revision 7853 by [[Special:Contributions/208.54.83.163|208.54.83.163]] ([[User talk:208.54.83.163|talk]])) [05:36:45] [[Query service/1]] ! 10https://wiki.toolserver.org/w/index.php?diff=7855&oldid=5727&rcid=21666 * 208.54.83.163 * (-3956) (Blanked the page) [05:39:54] Susan: ^ [05:40:08] [[Query service/1]] ! 10https://wiki.toolserver.org/w/index.php?diff=7856&oldid=7855&rcid=21667 * Legoktm * (+3956) (Undo revision 7855 by [[Special:Contributions/208.54.83.163|208.54.83.163]] ([[User talk:208.54.83.163|talk]])) [05:43:50] I saw. [05:44:15] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:208.54.83.16310]] with an expiry time of 1 week (anonymous users only): inexplicably blanking pages) [05:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:45:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12965.000000 [05:48:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12970 [05:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63915 MB (10% inode=99%): [05:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:53:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:59:36] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 934 MB (6% inode=99%): [06:14:36] / on wolfsbane is WARNING: DISK WARNING - free space: / 5055 MB (16% inode=92%): [06:14:36] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5054 MB (16% inode=92%): [06:15:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21082.000000 [06:20:36] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2135 [06:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 113964 MB (2% inode=66%): [06:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:32:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87650.000000 [06:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:45:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12022.000000 [06:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12020 [06:51:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63766 MB (10% inode=99%): [06:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:59:36] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 705 MB (4% inode=99%): [07:04:36] MySQL slave on z-dat-s2-b is OK: Uptime: 114696 Threads: 11 Questions: 189834776 Slow queries: 1717 Opens: 1348220 Flush tables: 1 Open tables: 256 Queries per second avg: 1655.112 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1728 [07:05:17] Load avg. on nightshade is WARNING: WARNING - load average: 19.24, 17.70, 15.23 [07:07:36] / on wolfsbane is OK: DISK OK - free space: / 7660 MB (25% inode=92%): [07:07:36] /tmp on ortelius is OK: DISK OK - free space: /tmp 4958 MB (33% inode=99%): [07:07:36] /tmp on wolfsbane is OK: DISK OK - free space: / 7660 MB (25% inode=92%): [07:10:15] Load avg. on nightshade is OK: OK - load average: 13.37, 14.96, 14.75 [07:15:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15126.000000 [07:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 113731 MB (2% inode=66%): [07:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:32:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85597.000000 [07:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:45:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11916.000000 [07:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11929 [07:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63573 MB (10% inode=99%): [07:52:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:15:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9700.000000 [08:26:16] Load avg. on nightshade is WARNING: WARNING - load average: 20.26, 19.21, 16.50 [08:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 113484 MB (2% inode=66%): [08:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:32:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81387.000000 [08:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9902.000000 [08:48:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9896 [08:51:16] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 25.23, 22.90, 20.28 [08:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63347 MB (10% inode=99%): [08:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:55:16] Load avg. on nightshade is WARNING: WARNING - load average: 17.72, 20.42, 19.89 [09:11:16] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 23.27, 22.12, 20.43 [09:14:15] Load avg. on nightshade is WARNING: WARNING - load average: 14.20, 19.28, 19.70 [09:15:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4286.000000 [09:23:36] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3514.000000 [09:28:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 113198 MB (2% inode=66%): [09:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:29:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:32:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78182.000000 [09:43:36] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1792.000000 [09:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:48:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8144.000000 [09:48:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8149 [09:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63122 MB (10% inode=99%): [09:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:57:35] why sge puts my tasks on willow, if using sge was needed to not overload willow? [10:08:16] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 33.81, 22.43, 18.73 [10:09:16] Load avg. on nightshade is WARNING: WARNING - load average: 22.41, 21.21, 18.54 [10:17:35] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2120 [10:17:56] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2123.000000 [10:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 112799 MB (2% inode=66%): [10:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:32:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76299.000000 [10:44:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9346.000000 [10:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9350 [10:51:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62871 MB (10% inode=99%): [10:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:07:36] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3609 [11:07:56] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [11:08:35] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3594 [11:08:56] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3581.000000 [11:09:16] Load avg. on nightshade is WARNING: WARNING - load average: 19.01, 17.58, 17.40 [11:12:36] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [11:12:56] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3614.000000 [11:25:37] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1934 MB (13% inode=99%): [11:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 112195 MB (2% inode=66%): [11:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:29:36] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1239 MB (9% inode=99%): [11:30:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:31:36] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1992 MB (14% inode=99%): [11:32:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76545.000000 [11:37:35] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4350 [11:37:56] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4351.000000 [11:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:46:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10218.000000 [11:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10227 [11:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62627 MB (10% inode=99%): [11:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:57:36] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1955 [12:09:16] Load avg. on nightshade is WARNING: WARNING - load average: 13.96, 14.64, 15.91 [12:20:15] Load avg. on nightshade is OK: OK - load average: 11.37, 13.01, 14.79 [12:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 110429 MB (2% inode=66%): [12:28:35] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:30:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:31:36] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1797 MB (12% inode=99%): [12:32:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78506.000000 [12:37:36] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5748 [12:37:56] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5734.000000 [12:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:44:36] MySQL slave on z-dat-s2-b is OK: Uptime: 135096 Threads: 6 Questions: 215967338 Slow queries: 1980 Opens: 1455624 Flush tables: 1 Open tables: 256 Queries per second avg: 1598.621 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1788 [12:46:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11441.000000 [12:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11446 [12:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62390 MB (10% inode=99%): [12:52:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:54:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:02:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50948 MB (5% inode=99%): [13:07:38] hello all [13:08:36] /tmp on ortelius is OK: DISK OK - free space: /tmp 3394 MB (23% inode=99%): [13:24:00] @replag [13:24:00] DaBPunkt: s1-user: 3h 25m 34s [+0.39 s/s]; s1-user-wd: 14m 25s [+0.28 s/s]; s2-rr: 23s [-0.73 s/s]; s2-user: 23s [-0.73 s/s]; s2-user-c: error; s2-user-wd: 21h 36m 31s [-0.13 s/s]; s3-user-wd: 9m 7s [+0.17 s/s]; s4-user-wd: 9m 12s [+0.17 s/s] [13:24:01] DaBPunkt: s5-user: 1h 53m 15s [+0.38 s/s]; s5-user-c: 2w 2d 15h 9m 59s [+0.10 s/s]; s6-user-wd: 8m 46s [+0.16 s/s]; s7-user-wd: 8m 31s [+0.15 s/s] [13:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 110868 MB (2% inode=66%): [13:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:30:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:32:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77589.000000 [13:33:36] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:36:36] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2874 MB (20% inode=99%): [13:37:36] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5779 [13:37:36] /tmp on ortelius is OK: DISK OK - free space: /tmp 3231 MB (22% inode=99%): [13:37:56] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5739.000000 [13:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:44:48] Susan: Thanks for debugging the issue. [13:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13130.000000 [13:48:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13135 [13:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62163 MB (10% inode=99%): [13:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:56:56] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3594.000000 [13:57:36] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3527 [14:13:36] MySQL slave on z-dat-s5-b is OK: Uptime: 136858 Threads: 5 Questions: 268908230 Slow queries: 1604 Opens: 134485 Flush tables: 1 Open tables: 256 Queries per second avg: 1964.870 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1791 [14:13:56] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1772.000000 [14:23:36] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2046 [14:23:56] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2032.000000 [14:27:57] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1783.000000 [14:27:57] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 110196 MB (2% inode=66%): [14:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:28:36] MySQL slave on z-dat-s5-b is OK: Uptime: 137758 Threads: 6 Questions: 269400309 Slow queries: 1624 Opens: 134657 Flush tables: 1 Open tables: 255 Queries per second avg: 1955.605 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1757 [14:30:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:32:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78795.000000 [14:33:16] Load avg. on nightshade is WARNING: WARNING - load average: 15.92, 15.24, 14.32 [14:33:36] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:35:13] 40% of the new enwiki-binary-dump are now prepared [14:35:16] Load avg. on nightshade is OK: OK - load average: 14.22, 14.95, 14.33 [14:44:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:46:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12657.000000 [14:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12634 [14:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61880 MB (10% inode=99%): [14:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:23:36] 60% of the new enwiki-binary-dump are now prepared [15:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 109687 MB (2% inode=66%): [15:28:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:30:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:32:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80224.000000 [15:33:36] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [15:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:48:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14338.000000 [15:48:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14345 [15:51:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61613 MB (10% inode=99%): [15:52:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:54:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:16:37] 84% of the new enwiki-binary-dump are now prepared [16:21:56] thats not over 9000% :p [16:22:16] what is a binary dump btw? isnt it always the database dump? [16:27:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 109269 MB (2% inode=66%): [16:28:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:30:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:32:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77876.000000 [16:33:36] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [16:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:48:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16229.000000 [16:48:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16230 [16:49:04] Maciej Jaros * [Toolserver-l] No keys/indexes on wiki databases? [16:51:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61303 MB (10% inode=99%): [16:52:16] / on ortelius is WARNING: DISK WARNING - free space: / 6236 MB (20% inode=92%): [16:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:54:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:07:04] John * Re: [Toolserver-l] No keys/indexes on wiki databases? [17:08:10] scfc_de: :-) [17:08:31] Danny_B: What's a binary dump? [17:13:48] ? [17:22:04] Tim Landscheidt * Re: [Toolserver-l] No keys/indexes on wiki databases? [17:22:57] ToAruShiroiNeko: Susan: A binary dump is the form the wmf provide us with. A normal dump is just a (big) text-file with (very) many INSERTs in it. A binary dump is more or less a copy of the complete database. It is faster to make and need no import, but is way bigger and need preparing before you can use it. [17:24:22] Danny_B: Oh, hah. Sorry, I'm just waking up and I completely misread the nick. :-) [17:27:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 108027 MB (2% inode=66%): [17:30:00] [[~arnomane/WP-autoreviewer]] !M 10https://wiki.toolserver.org/w/index.php?diff=7857&oldid=6645&rcid=21669 * Tim.Landscheidt * (-64) (Remove spam.) [17:30:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:32:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78810.000000 [17:33:36] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [17:41:43] [[Main Page/lang]] ! 10https://wiki.toolserver.org/w/index.php?diff=7858&oldid=7852&rcid=21670 * 66.54.89.123 * (+2) () [17:44:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60956 MB (9% inode=99%): [17:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:46:42] / on ortelius is WARNING: DISK WARNING - free space: / 5800 MB (19% inode=92%): [17:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17353.000000 [17:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17364 [17:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [17:51:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [17:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [17:51:22] [[~arnomane/WP-autoreviewer]] ! 10https://wiki.toolserver.org/w/index.php?diff=7859&oldid=7857&rcid=21671 * Tim.Landscheidt * (-30) (Remove wrong {{DEFAULTSORT}}.) [17:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1391697.000000 [17:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [17:51:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:53] /sql on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [17:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [17:52:13] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [17:54:12] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2910 [17:56:12] MySQL slave on z-dat-s4-a is OK: Uptime: 5165032 Threads: 1 Questions: 503897893 Slow queries: 283 Opens: 670 Flush tables: 11 Open tables: 225 Queries per second avg: 97.559 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1760 [18:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 107415 MB (2% inode=66%): [18:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78668.000000 [18:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60683 MB (9% inode=99%): [18:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:46:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:46:42] / on ortelius is WARNING: DISK WARNING - free space: / 5259 MB (17% inode=92%): [18:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17743.000000 [18:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17745 [18:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [18:51:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [18:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [18:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1393898.000000 [18:51:53] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [18:51:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [18:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [19:06:42] Load avg. on nightshade is WARNING: WARNING - load average: 23.49, 19.44, 16.13 [19:10:42] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 60.78, 31.28, 20.94 [19:10:53] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1909.000000 [19:22:42] / on ortelius is OK: DISK OK - free space: / 7892 MB (26% inode=92%): [19:30:42] Load avg. on nightshade is WARNING: WARNING - load average: 14.87, 17.79, 19.99 [19:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 107080 MB (2% inode=66%): [19:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78672.000000 [19:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60391 MB (9% inode=99%): [19:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:46:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17601.000000 [19:48:52] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1774.000000 [19:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17597 [19:51:05] Morten Wang * Re: [Toolserver-l] No keys/indexes on wiki databases? [19:51:13] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [19:51:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [19:51:23] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [19:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1396153.000000 [19:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [19:51:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [19:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [19:56:42] Load avg. on nightshade is OK: OK - load average: 12.97, 13.80, 14.95 [20:00:42] ok guys, enwiki is replicating on the re-setuped server. I will now look what is left of my wikipedia-free saturday. cu [20:34:52] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.402 second response time [20:36:52] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 14.515 second response time [20:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 106792 MB (2% inode=66%): [20:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:45:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76494.000000 [20:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59931 MB (9% inode=99%): [20:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [20:46:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17939.000000 [20:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17956 [20:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [20:51:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [20:51:24] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [20:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1397662.000000 [20:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [20:51:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [20:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [20:53:02] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 17.005 second response time [21:17:52] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1962.000000 [21:22:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:23:01] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 19.932 second response time [21:28:02] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 274 bytes in 20.110 second response time [21:45:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:45:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 106481 MB (1% inode=66%): [21:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75422.000000 [21:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59623 MB (9% inode=99%): [21:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [21:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:47:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:48:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19009.000000 [21:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19019 [21:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [21:51:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [21:51:24] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [21:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1399840.000000 [21:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [21:51:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [21:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [22:17:52] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2631.000000 [22:44:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 106117 MB (1% inode=66%): [22:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71727.000000 [22:46:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:46:05] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59261 MB (9% inode=99%): [22:46:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:46:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19400.000000 [22:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19410 [22:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [22:51:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [22:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [22:51:43] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1400680.000000 [22:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [22:51:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:53] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1779.000000 [22:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [22:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [22:56:02] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.913 second response time [22:58:02] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.795 second response time [22:59:02] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.960 second response time [23:00:52] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2122.000000 [23:03:17] [[Special:Log/newusers]] create 10 * Tebeeze * (New user account) [23:21:52] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1766.000000 [23:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 103102 MB (1% inode=66%): [23:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:45:57] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67603.000000 [23:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58940 MB (9% inode=99%): [23:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20589.000000 [23:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20582 [23:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [23:51:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [23:51:26] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [23:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1400017.000000 [23:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [23:51:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [23:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [23:55:28] nacht ts