[00:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 377360 MB (7% inode=35%): [00:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117802 MB (19% inode=99%): [00:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 39.64, 40.12, 40.20 [00:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:00:31] Free Memory on damiana is WARNING: WARNING - 6.1% (511364 kB) free! [01:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 376467 MB (7% inode=35%): [01:13:31] Free Memory on damiana is OK: OK - 7.2% (600260 kB) free. [01:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117608 MB (19% inode=99%): [01:23:07] does anyone know the format of the 'watchlist' table - what the columns mean? [01:25:30] The table is document on mediawiki.org. [01:25:40] dcoumented, rather. [01:25:44] God I can't type. :-( [01:25:56] carl-m: https://www.mediawiki.org/wiki/Manual:Watchlist_table [01:26:11] It's pretty straightforward. Any part in particular you have a question about? [01:26:29] The Toolserver has a restricted MySQL view of that table. [01:26:33] Susan: I mean the view of it that is in the toolserver replicas; mediawiki will not have a column named ts_wl_user_touched_cropped [01:26:33] Masking wl_user, for privacy reasons. [01:27:47] the contents of the column look like dates [01:28:16] Oh, that piece. [01:28:23] I'm not sure how well documented that is. [01:28:32] It was created by DaB. from a request by Dispenser, I think. [01:28:34] it's not, which is why I am asking here :) [01:28:38] There may be a JIRA ticket somewhere. [01:28:44] Or you can e-mail the list. [01:29:05] Asynchoronous communication seems to work better with the TS roots. I think they're the only ones who could look. [01:29:13] Unless Reedy still has access. [01:29:26] is it the case that the rows of the view correspond to the rows of the actual table - so that it is possible to count watchers by counting rows corresponding to a given title? [01:29:50] Yes. [01:29:57] I have a tool taht does that called "watcher". [01:30:13] I was the one who requested opening the watchlist table at all. Then Dispenser sought to have access expanded. [01:30:25] Every time a user watches a page, there are two rows added. [01:30:38] One for the subject-space page and one for the corresponding talk page. [01:30:56] that's the main thing I wanted to check [01:31:25] https://toolserver.org/~mzmcbride/watcher/ [01:32:02] https://github.com/mzmcbride/cgi-tools/blob/master/watcher.py [01:32:05] In case you're interested. [01:32:12] thanks [01:32:15] No problem. [01:32:35] If you want the view commands, just e-mail toolserver-l. Nosy or DaB. should be able to provide them. [01:37:32] Free Memory on damiana is WARNING: WARNING - 6.4% (533552 kB) free! [01:45:31] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422032 kB) free! [01:46:32] Free Memory on damiana is WARNING: WARNING - 5.2% (434884 kB) free! [01:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 43.88, 43.18, 42.91 [01:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 375518 MB (7% inode=35%): [02:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115699 MB (19% inode=99%): [02:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:50:21] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 91251 MB (9% inode=99%): [02:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:53:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.09, 41.78, 42.01 [02:53:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [03:01:22] does anyone have a copy of a lot of the hitcount data alrady downloaded into /mnt/user-store somewhere? [03:02:22] Check stats/? [03:02:37] That's the only place I know of. There's also an unhelpful README in the root of user-store, I think. [03:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:03:55] I am getting a huge 100KB/sec from dumps.wikimedia.org [03:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 374530 MB (7% inode=35%): [03:08:46] carl-m: Yeah, I filed a bug about that... [03:08:54] You may want to comment, let me find the link. [03:09:08] https://bugzilla.wikimedia.org/show_bug.cgi?id=43647 [03:13:20] I commented for you. [03:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117326 MB (19% inode=99%): [03:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 43.48, 44.14, 44.29 [03:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [04:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 372965 MB (7% inode=35%): [04:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117225 MB (19% inode=99%): [04:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.30, 44.11, 44.96 [04:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [05:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:08:10] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 369657 MB (6% inode=35%): [05:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117125 MB (19% inode=99%): [05:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 39.61, 41.45, 43.12 [05:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [06:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 367630 MB (6% inode=35%): [06:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117035 MB (19% inode=99%): [06:49:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 43.43, 43.30, 42.98 [06:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [06:59:31] Free Memory on damiana is WARNING: WARNING - 6.4% (536300 kB) free! [07:01:31] Free Memory on damiana is OK: OK - 7.4% (624040 kB) free. [07:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 362564 MB (6% inode=34%): [07:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116970 MB (19% inode=99%): [07:34:31] Free Memory on damiana is WARNING: WARNING - 6.0% (503408 kB) free! [07:45:31] Free Memory on damiana is OK: OK - 7.2% (603956 kB) free. [07:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 44.82, 45.75, 45.36 [07:53:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:54:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:54:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356517 MB (6% inode=34%): [08:16:31] Free Memory on damiana is WARNING: WARNING - 6.2% (523432 kB) free! [08:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116856 MB (19% inode=99%): [08:21:31] Free Memory on damiana is OK: OK - 7.3% (611284 kB) free. [08:49:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.91, 43.85, 44.46 [08:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:59:31] Free Memory on damiana is CRITICAL: CRITICAL - 3.2% (265644 kB) free! [09:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:03:31] Free Memory on damiana is OK: OK - 9.8% (817152 kB) free. [09:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 357383 MB (6% inode=34%): [09:19:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116649 MB (19% inode=99%): [09:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:34:41] / on ptolemy is WARNING: DISK WARNING - free space: / 4176 MB (20% inode=91%): [09:49:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:53:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.86, 37.57, 38.62 [09:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 354164 MB (6% inode=34%): [10:18:31] Free Memory on damiana is WARNING: WARNING - 5.7% (481592 kB) free! [10:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116393 MB (19% inode=99%): [10:27:31] /sql on cassia is WARNING: DISK WARNING - free space: /sql 98816 MB (8% inode=99%): [10:34:41] / on ptolemy is WARNING: DISK WARNING - free space: / 3992 MB (19% inode=91%): [10:36:31] Free Memory on damiana is OK: OK - 7.3% (607808 kB) free. [10:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:53:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.39, 40.63, 39.92 [10:53:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:54:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:04:31] Free Memory on damiana is WARNING: WARNING - 6.7% (561408 kB) free! [11:06:31] Free Memory on damiana is OK: OK - 7.1% (591212 kB) free. [11:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349713 MB (6% inode=33%): [11:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116165 MB (19% inode=99%): [11:34:40] / on ptolemy is WARNING: DISK WARNING - free space: / 3803 MB (19% inode=91%): [11:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.38, 38.43, 38.77 [11:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [12:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:08:10] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353944 MB (6% inode=34%): [12:08:31] Free Memory on damiana is WARNING: WARNING - 6.6% (553188 kB) free! [12:12:31] Free Memory on damiana is OK: OK - 7.2% (600588 kB) free. [12:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115919 MB (19% inode=99%): [12:34:41] / on ptolemy is WARNING: DISK WARNING - free space: / 3612 MB (18% inode=91%): [12:38:31] Free Memory on damiana is WARNING: WARNING - 6.8% (565760 kB) free! [12:40:31] Free Memory on damiana is OK: OK - 7.1% (596164 kB) free. [12:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:53:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.20, 38.83, 39.30 [12:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [13:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352384 MB (6% inode=34%): [13:17:31] Free Memory on damiana is WARNING: WARNING - 6.7% (564724 kB) free! [13:19:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115670 MB (19% inode=99%): [13:20:31] Free Memory on damiana is OK: OK - 7.2% (600232 kB) free. [13:28:30] Free Memory on damiana is WARNING: WARNING - 6.3% (527820 kB) free! [13:34:41] / on ptolemy is WARNING: DISK WARNING - free space: / 3402 MB (17% inode=91%): [13:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 39.66, 39.21, 39.19 [13:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [14:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 350125 MB (6% inode=33%): [14:09:31] Free Memory on damiana is OK: OK - 7.3% (608252 kB) free. [14:19:30] Free Memory on damiana is WARNING: WARNING - 6.6% (553628 kB) free! [14:19:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115431 MB (19% inode=99%): [14:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:34:41] / on ptolemy is WARNING: DISK WARNING - free space: / 3194 MB (15% inode=91%): [14:41:31] Free Memory on damiana is OK: OK - 7.3% (609860 kB) free. [14:48:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (364828 kB) free! [14:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:50:11] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:50:21] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 90307 MB (9% inode=99%): [14:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.09, 41.40, 40.29 [14:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:56:31] Free Memory on damiana is WARNING: WARNING - 5.7% (479972 kB) free! [15:00:35] I'm trying to use screen with a multi-maintainer project [15:01:01] In order for it to work, you need to run `ttyallow ` before becoming that project [15:01:13] But willow doesn't seem to have the `ttyallow` command [15:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:06:31] could i nudge anyone to take a look at an issue over two months old? TS-1560 [15:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348305 MB (6% inode=33%): [15:19:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115191 MB (18% inode=99%): [15:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:29:31] Free Memory on damiana is OK: OK - 7.2% (603704 kB) free. [15:34:41] / on ptolemy is WARNING: DISK WARNING - free space: / 2992 MB (14% inode=91%): [15:37:33] hello all [15:38:29] Hi DaBPunkt! [15:41:39] @replag [15:41:40] DaBPunkt: s1-rr-a-wd: 16s [-0.04 s/s]; s1-user-wd: 16s [-0.20 s/s]; s2-user: 28s [+0.00 s/s]; s2-user-c: 17s [-0.00 s/s]; s2-user-wd: 29s [-0.22 s/s]; s4-user-wd: 12s [-0.17 s/s]; s5-rr-a: 15s [-0.00 s/s]; s5-user: 15s [-0.00 s/s] [15:41:41] DaBPunkt: s5-user-c: 17s [-0.00 s/s]; s5-user-wd: 29s [-0.22 s/s]; s6-user-wd: 12s [-0.17 s/s]; s7-user-wd: 14s [-0.02 s/s] [15:41:49] good [15:42:31] Free Memory on damiana is WARNING: WARNING - 6.0% (505024 kB) free! [15:46:00] Do you know anything about using screen with MMPs? [15:48:20] wolfgang42: https://wiki.toolserver.org/view/MMP#Using_screen [15:48:41] on the linux-hosts it is just working AFAIK [15:48:50] (so no need for ttyallow) [15:49:16] Huh, so it is [15:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:49:27] * wolfgang42 goes to update the page [15:50:03] And this whole time I've been trying to get ttyallow to work. :P [15:50:11] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:51:11] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:51:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (401324 kB) free! [15:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 41.41, 41.65, 41.51 [15:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 346614 MB (6% inode=33%): [16:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115018 MB (18% inode=99%): [16:34:41] / on ptolemy is WARNING: DISK WARNING - free space: / 2779 MB (13% inode=91%): [16:38:41] / on ptolemy is OK: DISK OK - free space: / 7725 MB (38% inode=91%): [16:42:28] apmon_: can you please check if the renderd can log with another log-level than "notice"? [16:43:04] I'll have a look [16:43:47] I found the following in the apache doc: "When logging to a regular file messages of the level notice cannot be suppressed and thus are always logged. " [16:44:06] now I know why our erro-log is filling [16:44:09] do you mean mod_tile or renderd? [16:44:37] whatever "Thu Jan 03 21:57:55 2013] [notice] [client 10.24.1.24] Failed to connect to renderer" outputs [16:44:51] Those are from mod_tile [16:47:03] Hmm, if Apache isn't capable to suppress log level notice, then one would have to go through the code and comment out all of the error messages that are specified as notice [16:47:35] no, just replace it with "info" or "warn" [16:48:02] it → them [16:48:38] Well, either way, requires a search and replace and then recompile. Shouldn't be particularly difficult though [16:49:09] just can't currently do it in the next couple of days, as I am still out travelling [16:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:49:22] ok [16:49:51] Btw, do you occasionally clean up the tiles directory? [16:50:40] apmon_: I did somewhen at the beginning of Dec AFAIR. [16:50:49] should I do it now? [16:51:11] I am just wondering if there is a better way than is currently done [16:51:26] Are you deleting old files? [16:52:22] yes, with a detail-level>10 AFAIR [16:52:49] The way the system works is that the recently changed (and thus likely to be viewed) files are reset to a date in 2000. Which means the commonly viewed tiles are deleted. [16:53:04] It would be better if you use the access time rather than the modified time. [16:53:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:53:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 40.30, 41.59, 41.45 [16:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:53:51] "iles are reset to a date in 2000" why that? [16:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:54:17] It is the way to indicate which tiles have been modified and need re-rendering [16:55:06] so if they need to re-rendered anyway, why can I not delete them? [16:55:30] Because they can then not be used for caching until they are rerendered [16:56:03] Particularly on solaris, there is still an issue with on the fly rendering, that it doesn't work [16:56:31] So once some one views the tiles, they get rendered, but only are available the next time someone views them [16:56:45] the first time they get served from cache (if available) [16:57:43] and if I don't delete them the old data is displayed? [16:57:51] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:57:54] yes. [16:58:10] But often, only a small detail gets changed, so an old tile is still better than no tile [16:58:14] so wrong data vs. no-data ;) [16:58:36] ok, I will try to use atime next time [16:58:47] (as long as we update the atime) [16:58:51] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:58:57] and the expiry is overly aggressive as with out rendering it is difficult to determine if something has actually changed. So way more gets expired than actually visually changes [16:59:14] Cheers. Atime should hopefully work better. Thanks [17:00:38] Do you know if the file /tiles/planet-import-complete might have gotten deleted? [17:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351501 MB (6% inode=34%): [17:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114691 MB (18% inode=99%): [17:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.82, 43.19, 43.30 [17:53:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:54:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:59:20] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:08:10] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352590 MB (6% inode=34%): [18:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114494 MB (18% inode=99%): [18:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 41.56, 41.36, 41.59 [18:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:57:32] Free Memory on damiana is WARNING: WARNING - 4.6% (384108 kB) free! [18:59:21] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:00:21] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:03:32] Free Memory on damiana is OK: OK - 5.2% (432488 kB) free. [19:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351086 MB (6% inode=34%): [19:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114080 MB (18% inode=99%): [19:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 44.47, 42.88, 41.72 [19:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:03:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 355082 MB (6% inode=34%): [20:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113890 MB (18% inode=99%): [20:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:46:31] Free Memory on damiana is WARNING: WARNING - 4.6% (381816 kB) free! [20:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 41.63, 41.30, 41.42 [20:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:58:31] Free Memory on damiana is OK: OK - 5.2% (438252 kB) free. [21:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356277 MB (6% inode=34%): [21:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113668 MB (18% inode=99%): [21:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:48:31] Free Memory on damiana is WARNING: WARNING - 4.7% (395416 kB) free! [21:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:50:31] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 13.30, 26.26, 20.75 [21:51:31] Free Memory on damiana is OK: OK - 5.4% (455788 kB) free. [21:52:31] Load avg. on nightshade is WARNING: WARNING - load average: 11.59, 21.42, 19.64 [21:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:53:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 40.97, 41.75, 43.58 [21:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:55:31] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 59.18, 33.05, 23.88 [22:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:08:10] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 355034 MB (6% inode=34%): [22:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113528 MB (18% inode=99%): [22:27:31] /sql on cassia is WARNING: DISK WARNING - free space: /sql 97791 MB (8% inode=99%): [22:49:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.44, 42.82, 42.72 [22:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [23:03:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 357513 MB (6% inode=34%): [23:19:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113145 MB (18% inode=99%): [23:49:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:53:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 41.46, 42.68, 42.18 [23:53:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: