[00:02:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86140 MB (14% inode=99%): [00:05:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:08:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:10:47] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2042.000000 [00:11:18] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2058 [00:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:15:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:18:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:35:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17055.000000 [00:44:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:50:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:52:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:55:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:01:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:02:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86079 MB (14% inode=99%): [01:05:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:06:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [01:07:18] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3627 [01:08:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:16:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:18:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:29:26] Tim Landscheidt * Re: [Toolserver-l] How to have qsub mail output? [01:36:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17398.000000 [01:44:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:50:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:52:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:55:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:01:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:03:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85993 MB (14% inode=99%): [02:05:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:06:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5335.000000 [02:07:18] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5358 [02:08:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:16:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:36:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19051.000000 [02:44:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:50:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:53:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:55:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:00:28] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:01:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:02:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:03:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85928 MB (14% inode=99%): [03:06:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6821.000000 [03:07:18] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6827 [03:07:18] Free Memory on turnera is WARNING: WARNING - 6.6% (553140 kB) free! [03:08:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:16:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:18:18] Free Memory on turnera is OK: OK - 7.1% (599064 kB) free. [03:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:26:56] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Unavailable and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:36:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21192.000000 [03:43:07] /tmp on willow is WARNING: DISK WARNING - free space: / 21975 MB (20% inode=99%): [03:43:07] / on willow is WARNING: DISK WARNING - free space: / 21975 MB (20% inode=99%): [03:44:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:51:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:53:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:55:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:01:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:03:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:03:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85858 MB (14% inode=99%): [04:07:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6983.000000 [04:07:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6980 [04:09:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:12:03] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:25] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:19:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:27:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Unavailable and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:33:24] Free Memory on turnera is WARNING: WARNING - 5.7% (474868 kB) free! [04:36:25] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19751.000000 [04:43:12] / on willow is WARNING: DISK WARNING - free space: / 21882 MB (20% inode=99%): [04:43:12] /tmp on willow is WARNING: DISK WARNING - free space: / 21882 MB (20% inode=99%): [04:44:24] Free Memory on turnera is OK: OK - 7.2% (606052 kB) free. [04:44:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:51:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:53:43] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:55:33] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:03:42] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:04:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85833 MB (14% inode=99%): [05:07:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7149.000000 [05:08:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7146 [05:09:33] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:12:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:18:12] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:19:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:19:52] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:43] NTP on damiana is OK: NTP OK: Offset 0.001085 secs [05:27:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Unavailable and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5728.000000 [05:43:33] / on willow is WARNING: DISK WARNING - free space: / 21795 MB (20% inode=99%): [05:43:33] /tmp on willow is WARNING: DISK WARNING - free space: / 21795 MB (20% inode=99%): [05:44:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:48:31] Does anybody know if there's a plagiarism detection bot deployed to scan Wikipedia? [05:49:21] The English Wikipedia had a copyright violation bot. [05:49:33] And I think some people wanted to work with TurnItIn.com recently, maybe? [05:49:53] CorenSearchBot [05:50:26] Oh, right! It was down for a while. How soon I forget! [05:50:42] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3427.000000 [05:51:35] lexein: IIRC MadmanBot replaced CSB [05:51:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:52:48] Xlnt - looking forward to seeing if something can be cobbled together, at least at the recent edits feed! [05:54:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:56:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:57:54] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1644.000000 [06:02:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:04:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:04:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85786 MB (14% inode=99%): [06:08:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6377.000000 [06:08:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6388 [06:10:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:13:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:18:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:20:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:28:04] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Unavailable and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:44:33] / on willow is WARNING: DISK WARNING - free space: / 21705 MB (20% inode=99%): [06:44:33] /tmp on willow is WARNING: DISK WARNING - free space: / 21705 MB (20% inode=99%): [06:45:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:52:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:54:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:56:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:02:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:04:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:04:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85733 MB (14% inode=99%): [07:08:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5676.000000 [07:08:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5688 [07:10:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:13:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:19:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:20:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:28:04] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Unavailable and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:35:58] Hi. [07:36:35] hello [07:36:54] apmon: Are you able to tell me how the osm mapnik database is created exactly? I would like to create the same structure locally for testing [07:37:17] jongleur: Yes, I am [07:37:26] ;) great [07:37:37] that would be very nice [07:37:53] Do you have access to the ~osm/ directory on toolserver? [07:38:28] not sure... [07:38:46] let me see... [07:38:55] ~osm/tools/planet-import/planet-import is the script used to import the planet file [07:39:25] So it is a pretty standard import [07:39:40] apmon: cannot ls on ~osm at least [07:40:01] although I do manually create a few extra indexes afterwards [07:40:34] ah, a direct access to the planet-import dir works [07:41:20] well... extra indices are only to speed things up, at least they shouldn't produce additional problems when moving from local test to the toolserver [07:41:22] thanks [07:41:52] It is a completely standard import with --hstore-all by the looks of things [07:44:33] / on willow is WARNING: DISK WARNING - free space: / 21630 MB (20% inode=99%): [07:44:33] /tmp on willow is WARNING: DISK WARNING - free space: / 21630 MB (20% inode=99%): [07:44:37] apmon: which user runs that import? or is it the default style file? [07:45:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:45:15] I mean, it's still /osm2pgsql/default.style, but it's given explicitely in the import script, that's why I ask [07:46:09] it is the standard osm2pgsql style. Not sure when the special osm2pgsql got dropped that used to be used in the early days [07:46:23] k, thanks [07:52:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:54:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:57:03] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:02:33] / on willow is OK: DISK OK - free space: / 24481 MB (23% inode=99%): [08:02:33] /tmp on willow is OK: DISK OK - free space: / 24481 MB (23% inode=99%): [08:02:46] apmon: one last question: which osm2pgsql version is used on the toolserver? I'm not allowed to access the binary on the server - even for calling the help [08:03:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:03:36] jongleur: SVN version from a few days ago. Osm2pgsql is pretty much the only thing that is up-to-date on ptolemy though [08:03:45] ;) okay, thanks [08:04:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:04:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85657 MB (14% inode=99%): [08:05:52] * apmon needs to delete some stuff from ptolemy to stop it from echoing DISK WARNINGs all the time [08:08:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5664.000000 [08:11:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5692 [08:11:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:13:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:15:26] okay, import is running [08:15:44] thanks @ apmon so far [08:20:12] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:20:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:22:12] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@turnera-bge0 (using password: YES) [08:25:16] MySQL on ha-sql.esi is OK: Uptime: 4 Threads: 1 Questions: 1 Slow queries: 0 Opens: 15 Flush tables: 1 Open tables: 8 Queries per second avg: 0.250 [08:25:24] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:25:24] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:25:24] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:25:55] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:40:11] Hmmmmmm [08:40:33] willow is collapsed [08:40:53] Anybody at home? [08:48:30] I'm logged in to willow, but yes, I cannot do at least ls (not enough space for fork) [08:49:32] logging out there now.. [08:59:48] hello [08:59:56] head nodes error - working on it [09:13:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2595.000000 [09:13:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:13:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:14:26] seems to be back now [09:16:18] Yes, thanks :) [09:16:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 104563 MB (10% inode=99%): [09:18:26] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2917.000000 [09:18:32] @replag all [09:18:33] nosy: s1-rr-a: 48m 41s [-]; s1-user: 48m 41s [-]; s2-user: 22h 6m 47s [-]; s2-user-c: 48m 45s [-]; s3-rr-a: 48m 55s [-]; s3-user: 48m 55s [-]; s4-rr-a: 48m 45s [-]; s4-user: 48m 45s [-] [09:18:34] nosy: s5-rr-a: 48m 41s [-]; s5-user: 48m 41s [-]; s5-user-c: 48m 45s [-]; s6-rr-a: 48m 43s [-]; s6-user: 48m 43s [-]; s7-rr-a: 48m 43s [-]; s7-user: 48m 43s [-] [09:18:57] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2935.000000 [09:18:57] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2942.000000 [09:19:07] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2945.000000 [09:19:11] @replag all [09:19:11] nosy: s1-rr-a: 49m 20s [+1.00 s/s]; s1-user: 49m 20s [+1.02 s/s]; s2-user: 22h 7m 26s [+1.02 s/s]; s2-user-c: 49m 24s [+1.02 s/s]; s3-rr-a: 49m 34s [+1.02 s/s]; s3-user: 49m 34s [+1.02 s/s]; s4-rr-a: 49m 24s [+1.02 s/s]; s4-user: 49m 24s [+1.02 s/s] [09:19:12] nosy: s5-rr-a: 49m 20s [+1.02 s/s]; s5-user: 49m 20s [+1.02 s/s]; s5-user-c: 49m 24s [+1.02 s/s]; s6-rr-a: 49m 22s [+1.02 s/s]; s6-user: 49m 22s [+1.02 s/s]; s7-rr-a: 49m 21s [+1.00 s/s]; s7-user: 49m 21s [+1.00 s/s] [09:19:16] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2965.000000 [09:19:52] nosy: i've got "top: User name does not exist." when i was calling it on nightshade with my own name (sic!) [09:20:22] just now? [09:20:36] about 3 mins ago [09:21:30] ok my user is working [09:21:39] can you please try again? [09:24:15] Oct 5 09:21:15 nightshade nslcd[1318]: [80fe98] ldap_result() failed: Can't contact LDAP server [09:24:15] Oct 5 09:21:15 nightshade nslcd[1318]: [80fe98] ldap_abandon() failed to abandon search: Can't contact LDAP server [09:24:19] that might be the error [09:24:21] but [09:24:29] root@nightshade:/home/rnosy# telnet 10.24.1.11 ldap [09:24:29] Trying 10.24.1.11... [09:24:29] Connected to 10.24.1.11. [09:24:29] Escape character is '^]'. [09:25:22] logging in as my normal user and calling top works [09:25:32] works now [09:25:48] :) [09:26:21] so what is the cause of current issues? is it hw or sw based? [09:26:48] just a bad drive in damiana, one drive of two [09:27:19] but it made nginx service flap that was why it was in error state and no more websites from toolserver [09:27:39] i brought it online on turnera then and tried to offline damiana completely [09:27:54] * Damianz hands nosy fire [09:27:55] but wrong command...then everything hung [09:28:16] for a cigarette? ;) [09:28:43] Fire seems excessive for a cigarette. Though that sounds like a good idea. [09:29:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [09:30:06] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [09:30:16] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [09:30:27] @replag all [09:30:29] nosy: s1-rr-a: 58m 50s [+0.84 s/s]; s1-user: 58m 50s [+0.84 s/s]; s2-user: 22h 18m 42s [+1.00 s/s]; s2-user-c: 1h 40s [+1.00 s/s]; s3-rr-a: 58m 10s [+0.76 s/s]; s3-user: 58m 10s [+0.76 s/s]; s4-rr-a: 54m 42s [+0.47 s/s]; s4-user: 54m 42s [+0.47 s/s] [09:30:30] nosy: s5-rr-a: 58m 14s [+0.79 s/s]; s5-user: 58m 14s [+0.79 s/s]; s5-user-c: 1h 40s [+1.00 s/s]; s6-rr-a: 1h 14s [+0.96 s/s]; s6-user: 1h 14s [+0.96 s/s]; s7-rr-a: 52m 13s [+0.25 s/s]; s7-user: 52m 13s [+0.25 s/s] [09:31:07] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3254.000000 [09:32:26] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1476.000000 [09:36:17] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2104 [09:36:37] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2539 [09:36:50] @replag al [09:36:50] nosy: s1-rr-a: 41m 48s [-2.67 s/s]; s1-user: 41m 48s [-2.67 s/s]; s2-user: 22h 24m 29s [+0.91 s/s]; s2-user-c: 1h 3m 12s [+0.40 s/s]; s3-rr-a: 20m 50s [-5.85 s/s]; s3-user: 20m 50s [-5.85 s/s]; s5-user-c: 1h 3m 12s [+0.40 s/s]; s6-rr-a: 51m 13s [-1.41 s/s] [09:36:51] nosy: s6-user: 51m 13s [-1.41 s/s] [09:36:56] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3073 [09:37:57] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1776.000000 [09:38:16] @replag all [09:38:16] nosy: s1-rr-a: 39m 2s [-1.93 s/s]; s1-user: 39m 2s [-1.93 s/s]; s2-user: 22h 25m 9s [+0.46 s/s]; s2-user-c: 59m 10s [-2.81 s/s]; s3-rr-a: 9m 7s [-8.16 s/s]; s3-user: 9m 7s [-8.16 s/s]; s4-rr-a: 3s [-6.99 s/s]; s4-user: 3s [-6.99 s/s] [09:38:17] MySQL slave on thyme is OK: Uptime: 9034645 Threads: 11 Questions: 3020848957 Slow queries: 985357 Opens: 292641 Flush tables: 2 Open tables: 572 Queries per second avg: 334.362 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1625 [09:38:17] nosy: s5-rr-a: 3s [-7.45 s/s]; s5-user: 3s [-7.45 s/s]; s5-user-c: 59m 10s [-2.81 s/s]; s6-rr-a: 48m 14s [-2.08 s/s]; s6-user: 48m 14s [-2.08 s/s]; s7-rr-a: 5s [-6.68 s/s]; s7-user: 5s [-6.68 s/s] [09:38:56] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3418.000000 [09:44:57] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1729.000000 [09:46:36] MySQL slave on rosemary is OK: Uptime: 15186547 Threads: 45 Questions: 6739027463 Slow queries: 2149654 Opens: 301004 Flush tables: 6 Open tables: 4141 Queries per second avg: 443.749 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [09:46:57] MySQL slave on z-dat-s6-a is OK: Uptime: 434203 Threads: 14 Questions: 79656935 Slow queries: 27845 Opens: 728760 Flush tables: 1 Open tables: 2849 Queries per second avg: 183.455 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1689 [09:47:16] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1662.000000 [09:49:36] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71248 MB (7% inode=98%): [09:50:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38626 MB (9% inode=99%): [10:04:37] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71064 MB (7% inode=98%): [10:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85381 MB (14% inode=99%): [10:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:13:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:30:16] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7225.000000 [10:42:16] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3178.000000 [10:47:16] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1750.000000 [10:53:17] hello all [11:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85240 MB (14% inode=99%): [11:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:13:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:13:56] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:07] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:07] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:47] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 70286 MB (7% inode=98%): [11:15:47] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70286 MB (7% inode=98%): [11:16:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38540 MB (9% inode=99%): [11:31:16] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2195 [11:31:17] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2161 [11:34:16] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2141 [11:41:01] heyo [11:55:16] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [11:55:16] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [11:55:26] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3318 [11:55:37] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3613 [12:02:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3641 [12:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85123 MB (13% inode=99%): [12:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:13:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:27:07] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:37] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70122 MB (7% inode=98%): [12:54:06] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:54:06] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:54:36] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 70912 MB (7% inode=98%): [12:54:37] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70912 MB (7% inode=98%): [12:55:27] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6754 [12:56:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7114 [13:02:27] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6716 [13:09:17] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2125.000000 [13:09:36] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2143 [13:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84958 MB (13% inode=99%): [13:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:13:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:13:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:26:56] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:27:06] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:27:06] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:28:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38414 MB (9% inode=99%): [13:28:36] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71047 MB (7% inode=98%): [13:28:36] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71047 MB (7% inode=98%): [13:32:16] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [13:36:57] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1947.000000 [13:37:16] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3440.000000 [13:40:07] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:40:07] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:40:56] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:42:36] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3604 [13:43:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [13:43:37] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3515 [13:44:36] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3591 [13:45:36] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3606 [13:55:37] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [13:56:26] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9494 [13:58:36] MySQL slave on z-dat-s6-a is OK: Uptime: 449302 Threads: 13 Questions: 83136556 Slow queries: 33514 Opens: 787166 Flush tables: 1 Open tables: 2855 Queries per second avg: 185.34 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1696 [14:05:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3747.000000 [14:10:26] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3475 [14:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84772 MB (13% inode=99%): [14:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:13:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38296 MB (9% inode=99%): [14:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:13:36] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 70851 MB (7% inode=98%): [14:13:36] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70851 MB (7% inode=98%): [14:13:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:13:53] Is the user bryan here? [14:14:27] MySQL slave on z-dat-s7-a is OK: Uptime: 178459 Threads: 5 Questions: 54440864 Slow queries: 7965 Opens: 442146 Flush tables: 1 Open tables: 1529 Queries per second avg: 305.60 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1376 [14:16:55] @replag [14:16:56] Danny_B|backup: s1-rr-a: 1h 2m 28s [+0.08 s/s]; s1-user: 1h 2m 28s [+0.08 s/s]; s2-user: 1d 1h 40m 22s [+0.70 s/s]; s2-user-c: 48m 59s [-0.04 s/s]; s3-rr-a: 1h 15m 34s [+0.24 s/s]; s3-user: 1h 15m 34s [+0.24 s/s]; s5-user-c: 48m 59s [-0.04 s/s] [14:17:15] hmm, s3 is raising [14:21:47] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3474 [14:23:07] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3583.000000 [14:23:37] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3548 [14:28:47] MySQL slave on z-dat-s3-a is OK: Uptime: 103168 Threads: 20 Questions: 76200442 Slow queries: 25234 Opens: 1042936 Flush tables: 1 Open tables: 16384 Queries per second avg: 738.605 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1585 [14:30:10] what? one minute? [14:30:18] Danny_B|backup: ignore it [14:30:30] uff [14:30:34] I pushed enter to fast [14:30:40] :o) [14:30:56] should be "man shutdown", became "shutdown man" [14:30:57] good you did not push the enter too fast after rm -rf / ;-) [14:31:21] Danny_B|backup: that's my second worst nightmare… [14:31:29] it should be 2,5 more hours, right? [14:31:32] yes [14:31:47] hey guys, how long is willow gonna be down? I just logged in to run some queries. ;-) [14:31:59] tommorris: after 2,5 hrs [14:32:04] ah okay [14:32:06] it will be rebooted [14:32:13] see the list announce [14:32:27] I really ought to read my mailing list messages. ;-) [14:32:30] * Danny_B|backup sometimes thinks people ignore maillist completely [14:32:42] not to worry, I'll run the queries later [14:33:47] DaBPunkt: did you saw my email? I'm having problems in running a sh script from qcronsub on linux boxes [14:35:36] DaBPunkt: shouldn't it be 2.5 hrs? [14:35:51] utc is -2 from us [14:35:53] too much broadcast messages lately :p [14:36:07] Danny_B|backup: 16:00 UTC, it's 14:35 UTC now [14:36:24] we must noint to give DaBPunkt a nice present, on christmas :P [14:36:28] *joint [14:36:46] ah, i thought it was 1700 utc [14:36:57] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3047.000000 [14:37:26] great, I walled 2.5hs… [14:37:54] Alchimista: when did you send? [14:39:14] giftpflanze: yes, sorry [14:39:32] two or three days ago. Basicly, when sge startes him on linux boxes, it gives me an error like unable to open /bin/sh [14:41:18] title: Problem with qcronsub, linux boxes and a sh script [14:42:06] no, no mail here [14:45:58] nothing on toolserver-l@lists.wikimedia.org ? [14:46:28] Alchimista: you said you wrote an email to ME. That's the mailing-list [14:46:38] Ah, sorry :P [14:46:54] DaBPunkt: Is the /usr-mount /tiles thought to be stable again? Or are there further planned outages? [14:47:13] apmon: but I see no mail from 2-or-3-days-ago also on the ML [14:49:07] apmon: that depens what you call "stable". The current setup is a work-around and we hope it is stable (so no more downtimes). But we hope to fix the problem on hemlock so we can put it back where it belongs (so there will be another downtime) [14:49:16] afk [14:51:17] OK, thanks. That should be fine. [14:52:19] apmon: you can request fs-user-store on sge. if you script fails while runing just exit with code 99 and the job will be rescheduled when so mountoint is available again [14:53:56] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:54:06] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:54:07] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:54:17] Merlissimo: I don't really use sge. This is about serving osm map tiles on ptolemy which are stored on the NFS / SAN partition [14:54:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38248 MB (9% inode=99%): [14:54:37] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 70758 MB (7% inode=98%): [14:54:37] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70758 MB (7% inode=98%): [15:03:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3633.000000 [15:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84547 MB (13% inode=99%): [15:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:12:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:13:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:23:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3463.000000 [15:23:36] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3490 [15:24:33] who was responsible for dumps mmp again? [15:24:46] deleting "latest" dumps is stupid [15:25:05] theres no point in storing them in user-store in the first place, then [15:43:37] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3610 [15:44:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [15:55:33] @replag [15:55:33] DaBPunkt: s1-rr-a: 1h 7m 16s [+0.05 s/s]; s1-user: 1h 7m 16s [+0.05 s/s]; s2-user: 1d 3h 3m 26s [+0.84 s/s]; s2-user-c: 1h 20m 6s [+0.32 s/s]; s3-rr-a: 1m 2s [-0.76 s/s]; s3-user: 1m 2s [-0.76 s/s]; s5-user-c: 1h 20m 5s [+0.32 s/s] [16:02:51] 5 minutes for willow left. If you have open files close them NOW! [16:03:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4517.000000 [16:04:07] DaBPunkt: how long do you expect? [16:04:14] 5 minutes [16:04:41] ok, so i won't start my stuff on nightshade for backup, 5 mins is good [16:05:45] DaBPunkt: all existing process on willow will be killed right? [16:05:51] yes [16:06:32] ok [16:06:44] legoktm: can you please look at your bot at nightshade? It uses 100% of CPU-time [16:07:06] I just killed it [16:07:22] so how many users will disconnect ;-) [16:07:33] And I'll look into why its doing that now [16:09:11] DaBPunkt: Is there a way I can get a notification if my bot starts doing that again? [16:09:37] Sun Grid Engine execd on willow is CRITICAL: Connection refused by host [16:10:17] hmm, three only so far [16:10:35] my IRC bot died in another channel :P [16:10:41] Danny_B|webchat: not everyone misuses my servers as irc-gateways ;) [16:11:04] lat time there was reboot, about 10 users went down [16:11:44] ok, we are back [16:11:51] that was quick [16:11:56] and everything works? [16:12:00] yes, suprised me too [16:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84274 MB (13% inode=99%): [16:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:12:50] sge and puppet didn't start. That's no problem [16:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:14:59] ok guys. I will leave for making dinner (I'm already late) [16:15:01] afk [16:15:24] thanks DaBPunkt! [16:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:25:20] DaBPunkt: just fyi, the "become" command is not working for me on willow and hope you have a good dinner :) [16:43:37] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4021 [16:44:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4022.000000 [16:57:17] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:03:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5185.000000 [17:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83964 MB (13% inode=99%): [17:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:13:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:14:37] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3574 [17:15:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3538.000000 [17:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:38:43] DaBPunkt: Is the query killer running on sql-s1-user? I'm getting >300 sec before my query killer takes out /*LIMIT:1*/ queries [17:57:16] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:59:17] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:01:07] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1787.000000 [18:01:37] MySQL slave on rosemary is OK: Uptime: 15216248 Threads: 30 Questions: 6753013342 Slow queries: 2159532 Opens: 304217 Flush tables: 6 Open tables: 4138 Queries per second avg: 443.802 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1780 [18:03:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7423.000000 [18:05:26] Dispenser: I think I filed something in JIRA about that a while back [18:05:29] let me have a look [18:06:12] ah, no, that was something else, sorry [18:11:52] nvm. Looks like its just high load, the times aren't matching how my query killer is supposed to work [18:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83670 MB (13% inode=99%): [18:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:18:49] @replag [18:18:49] liangent: s1-rr-a: 27m 58s [-0.27 s/s]; s1-user: 27m 58s [-0.27 s/s]; s2-user: 1d 4h 32m 1s [+0.62 s/s]; s2-user-c: 2h 15m 48s [+0.39 s/s]; s5-user-c: 2h 15m 45s [+0.39 s/s] [18:20:52] re [18:26:50] Dispenser: yes, it is running. But the killer sleep after each round [18:42:45] DaBPunkt: You should announce it when you plan to reboot a login server [18:43:21] multichill: he did [18:43:29] multichill: http://lists.wikimedia.org/pipermail/toolserver-announce/2012-October/000529.html [18:44:49] Hmm, too much spam in that folder, dammit, I missed that one [18:48:36] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2039 [18:49:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2059.000000 [18:57:16] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:58:17] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:58:17] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:03:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9032.000000 [19:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83384 MB (13% inode=99%): [19:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:12:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:13:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:19:42] * MrAjedrez le pitan los oídos :) [19:48:37] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2164 [19:49:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2116.000000 [20:03:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9065.000000 [20:07:37] MySQL slave on rosemary is OK: Uptime: 15223807 Threads: 20 Questions: 6755925241 Slow queries: 2161779 Opens: 304792 Flush tables: 6 Open tables: 4140 Queries per second avg: 443.773 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1789 [20:08:06] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1772.000000 [20:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83142 MB (13% inode=99%): [20:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:13:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:15:36] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2007 [20:16:07] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2031.000000 [20:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:26:37] MySQL slave on rosemary is OK: Uptime: 15224947 Threads: 13 Questions: 6756618477 Slow queries: 2161952 Opens: 304909 Flush tables: 6 Open tables: 4141 Queries per second avg: 443.786 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [20:27:06] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1728.000000 [21:03:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5652.000000 [21:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82969 MB (13% inode=99%): [21:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:12:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:16:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 104457 MB (10% inode=99%): [21:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:03:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4362.000000 [22:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82754 MB (13% inode=99%): [22:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:13:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:17:57] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3525.000000 [22:26:57] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1771.000000 [23:00:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:00:34] ^ this doesn't look good... [23:02:49] argl [23:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82517 MB (13% inode=99%): [23:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:12:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:20:58] nacht ts [23:21:43] gn8 ts