[00:23:28] Hummmmmmm [00:23:34] Someone awake? [00:24:10] Just ask your question. [00:25:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:25:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42415 MB (4% inode=99%): [00:26:22] Sun Grid Engine execd on willow is CRITICAL: all.q@willow in unknown state: longrun@willow in unknown state [00:26:41] Cluster on turnera.esi is CRITICAL: damiana FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote damiana Offline, mysql OFFLINE, check nfs-hasp, nfs-home Online, [00:27:01] Free Memory on damiana.esi is OK: OK - 87.6% (3665468 kB) free. [00:27:01] MySQL on ha-sql.esi is CRITICAL: Cant connect to MySQL server on ha-sql.esi (146) [00:27:22] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:27:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (1 error): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_SAS_PORT_DEGRADED.description:S27:Tray.85.Controller.A.Port.3: [00:29:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:29:43] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [00:29:43] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [00:30:02] MySQL on ha-sql.esi is OK: Uptime: 56 Threads: 5 Questions: 1367 Slow queries: 1 Opens: 24 Flush tables: 1 Open tables: 13 Queries per second avg: 24.410 [00:30:43] CAM on hemlock is OK: OK - cam detected no new errors [00:31:32] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [00:31:42] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [00:33:07] Joan: I was just going to comment that the shell servers were having problems, but obviously it's solved now :) [00:33:22] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.325684/1.75, alarm hl:np_load_avg=0.283691/2.00, alarm hl:mem_free=265.000000M/300M [00:48:14] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1929.000000 [00:48:23] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1946.000000 [00:48:23] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1937.000000 [00:48:33] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1944.000000 [00:48:43] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1952.000000 [01:16:13] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [01:16:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [01:16:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [01:16:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3633.000000 [01:17:23] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 329.000000 [01:17:32] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3592.000000 [01:18:13] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3580.000000 [01:18:24] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3569 [01:18:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628 [01:18:42] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2432 [01:18:52] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3582 [01:18:52] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2859 [01:19:02] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3095 [01:20:42] MySQL slave on z-dat-s3-a is OK: Uptime: 14605 Threads: 20 Questions: 6746419 Slow queries: 211 Opens: 45415 Flush tables: 1 Open tables: 16384 Queries per second avg: 461.925 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1128 [01:20:52] MySQL slave on z-dat-s7-a is OK: Uptime: 139692 Threads: 15 Questions: 12957969 Slow queries: 5542 Opens: 60939 Flush tables: 1 Open tables: 3154 Queries per second avg: 92.760 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1672 [01:21:24] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3592.000000 [01:21:43] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3579 [01:24:02] MySQL slave on z-dat-s6-a is OK: Uptime: 200260 Threads: 16 Questions: 16936253 Slow queries: 11650 Opens: 142243 Flush tables: 1 Open tables: 1790 Queries per second avg: 84.571 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1652 [01:24:24] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [01:24:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18155.000000 [01:24:24] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [01:24:24] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:24:24] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29429.000000 [01:25:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43181 MB (4% inode=99%): [01:25:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:28:13] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1729.000000 [01:28:24] MySQL slave on daphne is OK: Uptime: 6489053 Threads: 45 Questions: 5370034569 Slow queries: 842748 Opens: 35149517 Flush tables: 3 Open tables: 16367 Queries per second avg: 827.552 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1633 [01:29:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:42:33] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1752.000000 [01:42:53] MySQL slave on rosemary is OK: Uptime: 8062626 Threads: 14 Questions: 2416947243 Slow queries: 1010754 Opens: 12050 Flush tables: 1 Open tables: 1014 Queries per second avg: 299.771 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1718 [01:48:42] MySQL slave on thyme is OK: Uptime: 3014294 Threads: 14 Questions: 1043014386 Slow queries: 517265 Opens: 149061 Flush tables: 1 Open tables: 2796 Queries per second avg: 346.22 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1743 [01:49:24] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1510.000000 [02:16:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7240.000000 [02:19:33] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.550293/1.75, alarm hl:np_load_avg=0.673828/2.00, alarm hl:mem_free=196.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.550293/1.50, alarm hl:np_load_long=0.595703/1.75, alarm hl:mem_free=196.000000M/250M [02:25:24] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [02:25:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43061 MB (4% inode=99%): [02:25:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21579.000000 [02:25:24] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [02:25:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:25:24] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:25:24] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32970.000000 [02:28:32] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:30:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:31:54] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3329.000000 [02:32:32] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.838379/1.75, alarm hl:np_load_avg=0.806641/2.00, alarm hl:mem_free=272.000000M/300M [02:34:55] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1268.000000 [03:25:33] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14161.000000 [03:26:23] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [03:26:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42935 MB (4% inode=99%): [03:26:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13125.000000 [03:26:24] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [03:26:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:26:24] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:30:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:44:32] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2174.000000 [03:45:32] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 9.000000 [03:49:23] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3474.000000 [03:52:23] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 962.000000 [04:04:15] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.332031/1.00, alarm hl:np_load_long=0.897461/1.50, alarm hl:mem_free=21471.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.332031/1.10, alarm hl:np_load_long=0.897461/1.75, alarm hl:mem_free=21471.000000M/300M [04:12:13] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [04:26:33] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [04:26:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:26:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:27:23] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [04:27:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42794 MB (4% inode=99%): [04:30:34] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:34:15] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.209961/1.00, alarm hl:np_load_long=1.005859/1.50, alarm hl:mem_free=21348.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.209961/1.10, alarm hl:np_load_long=1.005859/1.75, alarm hl:mem_free=21348.000000M/300M [04:40:13] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [04:46:14] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.112305/1.00, alarm hl:np_load_long=1.028320/1.50, alarm hl:mem_free=21003.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.112305/1.10, alarm hl:np_load_long=1.028320/1.75, alarm hl:mem_free=21003.000000M/300M [04:50:14] Load avg. on ortelius is WARNING: WARNING - load average: 16.57, 12.36, 7.64 [04:51:15] Load avg. on ortelius is OK: OK - load average: 8.76, 10.97, 7.45 [04:59:14] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [05:04:14] Free Memory on turnera.esi is WARNING: WARNING - 12.7% (530232 kB) free! [05:26:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:27:32] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [05:27:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:28:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42526 MB (4% inode=99%): [05:28:22] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [05:30:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:34:14] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.069336/1.00, alarm hl:np_load_long=1.014649/1.50, alarm hl:mem_free=22332.000000M/300M [05:35:13] Free Memory on turnera.esi is CRITICAL: CRITICAL - 9.7% (405488 kB) free! [05:37:14] Free Memory on turnera.esi is WARNING: WARNING - 10.3% (431512 kB) free! [05:37:15] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [05:45:14] Free Memory on turnera.esi is CRITICAL: CRITICAL - 10.0% (418696 kB) free! [05:45:14] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.487305/1.00, alarm hl:np_load_long=1.031250/1.50, alarm hl:mem_free=22238.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.487305/1.10, alarm hl:np_load_long=1.031250/1.75, alarm hl:mem_free=22238.000000M/300M [06:02:35] is it purposeful to have http://status.toolserver.org return a "Internal Server Error" ? [06:03:22] <- trying to find the status of the recent fix job as I am needing to know when to ask Bryan to restart CommandsDelinker bot [06:06:35] erm, i think river hosted that, and she ain't been seen for a while [06:15:16] any knowledge on the server fix? [06:19:35] ask dab. or nosy [06:19:59] sDrewth: i guess it's being fixed http://lists.wikimedia.org/pipermail/toolserver-l/2012-February/004711.html [06:26:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:27:32] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [06:27:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:28:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42571 MB (4% inode=99%): [06:28:23] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [06:30:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:44:14] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.081055/1.00, alarm hl:np_load_long=0.764648/1.50, alarm hl:mem_free=22332.000000M/300M [06:45:13] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [06:50:23] zzz [07:12:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.758789/1.75, alarm hl:np_load_avg=0.790527/2.00, alarm hl:mem_free=244.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.758789/1.50, alarm hl:np_load_long=0.738769/1.75, alarm hl:mem_free=244.000000M/250M [07:13:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:26:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:27:32] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [07:27:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:29:23] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [07:29:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42448 MB (4% inode=99%): [07:31:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:51:20] Free Memory on turnera.esi is CRITICAL: CRITICAL - 9.0% (378568 kB) free! [08:15:03] Kolossos * Re: [Toolserver-l] External authentication? [08:22:54] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 16529 MB (1% inode=99%): [08:26:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:27:32] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [08:28:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:29:33] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [08:29:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42332 MB (4% inode=99%): [08:31:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:37:03] Hi. What happened to /mnt/user-store/stats? The files used to be downloaded there automatically, but there are no stats from 2012 at all [08:49:14] Load avg. on adenia is WARNING: WARNING - load average: 19.63, 11.99, 7.31 [08:51:15] Load avg. on adenia is OK: OK - load average: 12.21, 12.25, 8.03 [08:52:20] Free Memory on turnera.esi is CRITICAL: CRITICAL - 5.2% (219028 kB) free! [08:59:14] Free Memory on turnera.esi is WARNING: WARNING - 11.1% (464004 kB) free! [09:04:13] Free Memory on turnera.esi is CRITICAL: CRITICAL - 10.0% (419264 kB) free! [09:05:14] Free Memory on turnera.esi is WARNING: WARNING - 10.1% (422712 kB) free! [09:06:24] johang, I found a few pagecounts-* files in your folder on /mnt/user-store. I can see that you download them daily. Wouldn't it be wiser to download them to /mnt/user-store/stats, where all the other pagecounts are? [09:26:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:27:02] Adam Klimont * [Toolserver-l] What happened to pagecounts stats? [09:27:33] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [09:28:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:29:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43112 MB (4% inode=99%): [09:29:33] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [09:31:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:36:13] Free Memory on turnera.esi is CRITICAL: CRITICAL - 6.5% (273808 kB) free! [09:38:43] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1924.000000 [09:39:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.478027/1.75, alarm hl:np_load_avg=0.464844/2.00, alarm hl:mem_free=231.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.478027/1.50, alarm hl:np_load_long=0.474121/1.75, alarm hl:mem_free=231.000000M/250M [09:47:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:50:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.478027/1.75, alarm hl:np_load_avg=0.496582/2.00, alarm hl:mem_free=282.000000M/300M [10:08:35] Can disabled tools like http://toolserver.org/~soxred93/pcount/index.php be saved by the community or looked to be moved somewhere stable at WMF? [10:09:21] 3(created) [MAGNUS-297] https support for https://toolserver.org/~magnus/file_siblings.php; Magnus' tools; Minor Support request <10https://jira.toolserver.org/browse/MAGNUS-297> (Umherirrender) [10:26:44] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:28:33] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [10:28:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:30:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42981 MB (4% inode=99%): [10:30:32] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [10:31:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:36:23] Free Memory on turnera.esi is CRITICAL: CRITICAL - 3.0% (126056 kB) free! [10:38:44] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2993.000000 [10:59:44] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [11:03:44] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1878.000000 [11:07:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.445801/1.75, alarm hl:np_load_avg=0.405274/2.00, alarm hl:mem_free=251.000000M/300M [11:11:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [11:27:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:28:32] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [11:28:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:30:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42828 MB (4% inode=99%): [11:30:32] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [11:31:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:36:22] Free Memory on turnera.esi is CRITICAL: CRITICAL - 2.3% (96184 kB) free! [11:49:43] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3616.000000 [12:03:35] hello all [12:04:35] Hi, DaBPunkt [12:05:04] It seems there are problems again [12:05:18] I look fort it now [12:05:28] Ok, thanks :) [12:13:49] It loooks like it takes a moment [12:17:33] turnera is down/unreachable [12:20:18] DaBPunkt: TS is offline as it seems :/ [12:20:35] I know. It is s problem with the HA-cluster [12:43:44] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3206.000000 [12:43:44] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3206.000000 [12:43:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42816 MB (4% inode=99%): [12:44:22] ok, should be fine again [12:44:33] Yes, seems to work :) [12:44:44] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3265.000000 [12:44:54] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3278.000000 [12:45:13] Cluster on damiana.esi is CRITICAL: turnera FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote turnera Offline, check nfs-hasp, nfs-home Online, [12:45:14] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3300.000000 [12:45:33] Free Memory on turnera.esi is OK: OK - 87.3% (3652432 kB) free. [12:47:04] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [12:47:53] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/fmd:default svc:/system/cluster/scsymon-srv:default svc:/system/dumpadm:default [12:49:14] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [12:50:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [12:50:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [12:50:44] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [12:50:53] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3639.000000 [12:51:14] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3660.000000 [12:55:34] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2877 [12:56:54] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1299.000000 [12:57:44] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3454.000000 [12:57:44] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3435.000000 [12:58:13] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3284 [12:58:23] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3307 [12:58:44] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2720 [12:58:54] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3864 [13:01:33] MySQL slave on z-dat-s6-a is OK: Uptime: 242112 Threads: 21 Questions: 21602946 Slow queries: 12740 Opens: 148591 Flush tables: 1 Open tables: 1790 Queries per second avg: 89.227 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1737 [13:02:44] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3592.000000 [13:02:54] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3544 [13:03:13] MySQL slave on thyme is OK: Uptime: 3054765 Threads: 13 Questions: 1051453562 Slow queries: 518933 Opens: 150469 Flush tables: 1 Open tables: 2792 Queries per second avg: 344.201 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1710 [13:03:23] MySQL slave on rosemary is OK: Uptime: 8103454 Threads: 8 Questions: 2426733287 Slow queries: 1012125 Opens: 12077 Flush tables: 1 Open tables: 1015 Queries per second avg: 299.469 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1673 [13:03:44] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1589.000000 [13:03:44] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1551.000000 [13:03:44] MySQL slave on z-dat-s7-a is OK: Uptime: 181861 Threads: 21 Questions: 18392269 Slow queries: 6402 Opens: 123286 Flush tables: 1 Open tables: 3189 Queries per second avg: 101.133 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1703 [13:05:03] DaB. * Re: [Toolserver-l] FYI: login.toolserver not responding or asking for password, HTTP down [13:06:34] Free Memory on damiana.esi is WARNING: WARNING - 18.3% (765844 kB) free! [13:08:14] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [13:08:14] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [13:28:11] MySQL slave on daphne is OK: Uptime: 6532239 Threads: 52 Questions: 5396196210 Slow queries: 854917 Opens: 35304953 Flush tables: 3 Open tables: 16379 Queries per second avg: 826.86 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1788 [13:29:11] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1731.000000 [13:29:31] Free Memory on damiana.esi is OK: OK - 12.5% (524316 kB) free. [13:43:11] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [13:43:11] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11518.000000 [13:43:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:43:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:43:11] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [13:43:12] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10132.000000 [13:44:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42620 MB (4% inode=99%): [13:48:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:49:11] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6623 [13:52:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7320.000000 [14:01:11] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3575.000000 [14:05:12] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1507.000000 [14:21:12] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2945.000000 [14:24:11] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1368.000000 [14:39:11] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3569.000000 [14:43:11] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [14:44:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42459 MB (4% inode=99%): [14:44:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:44:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:44:11] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [14:48:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:49:11] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [15:09:12] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1734.000000 [15:17:11] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3227 [15:20:11] MySQL slave on z-dat-s3-a is OK: Uptime: 64973 Threads: 19 Questions: 73848520 Slow queries: 1904 Opens: 220218 Flush tables: 1 Open tables: 16384 Queries per second avg: 1136.603 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1498 [15:29:41] Free Memory on damiana.esi is WARNING: WARNING - 7.9% (331128 kB) free! [15:35:41] Free Memory on damiana.esi is OK: OK - 10.4% (434964 kB) free. [15:43:11] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [15:44:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42296 MB (4% inode=99%): [15:44:11] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [15:44:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:44:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:46:41] Free Memory on damiana.esi is WARNING: WARNING - 9.6% (401728 kB) free! [15:48:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:57:41] Free Memory on damiana.esi is CRITICAL: CRITICAL - 5.8% (241668 kB) free! [16:00:26] Anyone else not able to connect to the ts? [16:00:51] (I'm trying to connect to http://svn.toolserver.org/ and http://toolserver.org/~cvn/) [16:02:07] is toolserver working to anyone? [16:02:49] via http => nothing, apparently. via ssh => it asks a password which shouldn't be necessary, doesn't allow to connect [16:02:57] please wait a moment [16:03:03] ok [16:03:09] joancreus, I'm having trouble w/ svn and http as well [16:03:44] TsLogBot has a ping timeout, suppose it can be confirmed [16:04:22] teh chanserver is NOt my faiul! [16:04:24] ;) [16:04:39] :) a global notice appeared some minutes ago [16:04:59] * joancreus DaBPunkt has broken freenode! [16:05:11] * Firebolt hits DaBPunkt with a broom [16:05:14] you broke chanserv! [16:05:18] Hmmmmm [16:05:58] Confabulation of Toolserver and Freenode? :) [16:06:23] They're starting their world takeover plan [16:08:10] damiana rebots now, everything should be back to normal in 5 minutes [16:08:20] thanks DaBPunkt ! [16:10:47] DaBPunkt: ssh now working fine [16:10:49] thanks [16:10:51] Cluster on damiana.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, nfs-home Online, [16:11:02] DiskSuite on turnera.esi is CRITICAL: CRITICAL - submirror d11 of mirror d10 is Resyncing and submirror d12 of mirror d10 is Resyncing [16:11:17] ok, I reboot the other box (turnera) now. Nothing bad SHOULD happen [16:12:22] Cluster on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:12:26] can wikiquote (the catalan one, to be specific) be accessed via mysql on commandline? [16:12:38] joancreus: sure [16:13:06] DaBPunkt: server/table? thanks [16:13:26] joancreus: dab@nightshade:~$ sql -r cawikiquote_p [16:13:38] ok. thanks a lot [16:13:52] Cluster on turnera.esi is OK: CLUSTER OK ! [16:13:55] (we have a wiki with howtos to ;)) [16:14:01] to → too [16:14:10] https://wiki.toolserver.org/view/Database_access ? [16:14:39] yes [16:14:51] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 waiting, check nfs-hasp, [16:15:02] DiskSuite on turnera.esi is OK: OK - No disk failures detected [16:15:44] is it just me or is the "_" in "$ sql enwiki_p" mising on the wiki-page? [16:15:52] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [16:16:22] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.003869 secs [16:16:52] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1953.000000 [16:19:11] [[Database access]] 10https://wiki.toolserver.org/w/index.php?diff=6654&oldid=6424&rcid=8780 * Dab * (-16) (/* Command-line access */ Not helpfull if the "_" vanish) [16:30:21] NTP on turnera.esi is OK: NTP OK: Offset -0.016389 secs [16:43:32] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [16:44:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42549 MB (4% inode=99%): [16:44:21] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:44:52] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [16:44:52] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3632.000000 [16:44:53] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:48:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:13:52] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.697266/1.75, alarm hl:np_load_avg=0.717285/2.00, alarm hl:mem_free=270.000000M/300M [17:15:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [17:17:50] ts-admins: Can I get an update on https://jira.toolserver.org/browse/TS-1285 please? I'm still gettign spammed constantly about my expired tools. [17:30:21] Free Memory on damiana.esi is WARNING: WARNING - 9.6% (401064 kB) free! [17:32:22] 3(resolved) [TS-1260] Install MySQL Spatial extensions <10https://jira.toolserver.org/browse/TS-1260> (Marlen Caemmerer) [17:34:22] 3(commented) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (Marlen Caemmerer) [17:35:22] Free Memory on damiana.esi is OK: OK - 10.2% (426928 kB) free. [17:38:23] 3(assigned) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Marlen Caemmerer) [17:40:21] 3(commented) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Marlen Caemmerer) [17:40:22] Free Memory on damiana.esi is WARNING: WARNING - 9.7% (406968 kB) free! [17:43:32] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [17:44:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42957 MB (4% inode=99%): [17:44:21] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:44:51] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [17:44:51] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7232.000000 [17:44:51] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:45:22] 3(commented) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (Marlen Caemmerer) [17:48:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:07:00] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1943 [18:08:20] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1932 [18:12:21] 3(resolved) [MAGNUS-297] https support for https://toolserver.org/~magnus/file_siblings.php <10https://jira.toolserver.org/browse/MAGNUS-297> (Magnus Manske) [18:12:50] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.686523/1.75, alarm hl:np_load_avg=0.682129/2.00, alarm hl:mem_free=276.000000M/300M [18:13:50] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [18:14:21] 3(commented) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Soxred93) [18:15:20] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.1% (172708 kB) free! [18:20:50] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1909 [18:43:50] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [18:45:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42803 MB (4% inode=99%): [18:45:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:45:50] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [18:45:50] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8085.000000 [18:45:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:49:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:53:20] MySQL slave on z-dat-s6-a is OK: Uptime: 263212 Threads: 9 Questions: 23761575 Slow queries: 16181 Opens: 148674 Flush tables: 1 Open tables: 1828 Queries per second avg: 90.275 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1741 [18:56:50] MySQL slave on z-dat-s3-a is OK: Uptime: 77974 Threads: 25 Questions: 80693576 Slow queries: 4482 Opens: 248694 Flush tables: 1 Open tables: 16384 Queries per second avg: 1034.877 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1631 [19:07:00] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2684 [19:12:51] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.728027/1.75, alarm hl:np_load_avg=0.659668/2.00, alarm hl:mem_free=284.000000M/300M [19:13:00] MySQL slave on z-dat-s7-a is OK: Uptime: 204022 Threads: 2 Questions: 20040828 Slow queries: 8790 Opens: 123344 Flush tables: 1 Open tables: 3244 Queries per second avg: 98.228 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1691 [19:13:51] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [19:15:20] Free Memory on damiana.esi is CRITICAL: CRITICAL - 3.3% (137712 kB) free! [19:33:16] and it's down [19:33:58] http://www.isup.me/toolserver.org [19:43:50] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [19:45:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42634 MB (4% inode=99%): [19:45:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:45:51] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [19:45:51] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8119.000000 [19:46:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:49:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:54:07] mono: site is up from my end [19:54:33] it's being intermittent [19:54:53] http://status.toolserver.org/ is broken [19:59:09] mono: send email to webmaster@RT.UK.EU.ORG :) [20:00:18] already done to TS emergency address [20:14:29] Everything is down again... [20:14:41] zzzzzzzzz [20:14:44] DaBPunkt: Are you aware ? [20:20:12] nosy1: ? [20:24:01] Free Memory on damiana.esi is OK: OK - 52.9% (2212132 kB) free. [20:24:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:24:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42595 MB (4% inode=99%): [20:25:10] LDAP on ha-ldap.esi is CRITICAL: Could not bind to the LDAP server [20:25:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:25:41] Cluster on damiana.esi is CRITICAL: ldap OFFLINE, check nfs-hasp, ds--global-misc-ldap OFFLINE, [20:26:11] NTP on damiana.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.008507 secs [20:26:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:26:11] Cluster on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:26:21] NTP on turnera.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:40] s5 replag on daphne is CRITICAL: (Service Check Timed Out) [20:27:00] hello. I can no more connect to Toolserver ("server refused our key"). Does somebody know what to do ? [20:27:35] Quentinv57, there are general problems at the moment [20:27:52] jem-, okay, so that's not my fault ? [20:28:00] Very probably not :) [20:28:11] It seems a server is getting its memory full for unknown causes [20:28:20] LDAP on ha-ldap.esi is WARNING: LDAP WARNING - 6.332 seconds response time [20:28:30] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [20:28:34] (Or that's what they told to the mailing list) [20:28:40] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [20:28:40] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [20:29:01] Sun Grid Engine execd on wolfsbane is OK: short@wolfsbane OK: all.q@wolfsbane OK [20:29:06] There have been 4-5 interruptions in these 2 days [20:29:11] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.005321 secs [20:29:11] LDAP on ha-ldap.esi is OK: LDAP OK - 0.019 seconds response time [20:29:51] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [20:31:26] okay jem- thanks for the info [20:33:11] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1939.000000 [20:33:21] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1945.000000 [20:33:21] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1947.000000 [20:33:41] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1961.000000 [20:33:41] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1965.000000 [20:34:25] Np, Quentinv57 :) [20:34:36] It seems it's back to normal [20:34:52] Quentinv57, you should be able to login now [20:35:39] jem-, yes, everything is okay now, thanks [20:35:53] but also my bot has automatically restarted, so I no more need to do it :P [20:37:07] :)) [20:39:10] NTP on damiana.esi is OK: NTP OK: Offset -0.01768 secs [20:40:11] s2 replag on daphne is CRITICAL: QUERY CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [20:42:11] MySQL on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [20:42:21] MySQL slave on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [20:44:11] NTP on turnera.esi is OK: NTP OK: Offset -0.003227 secs [20:50:04] why isn't fist working? [21:01:11] MySQL on daphne is OK: Uptime: 6559423 Threads: 47 Questions: 5409302563 Slow queries: 862652 Opens: 35400968 Flush tables: 3 Open tables: 16362 Queries per second avg: 824.661 [21:01:41] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000 [21:02:20] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 0.000000 [21:02:20] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3599 [21:03:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3632 [21:03:11] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3559.000000 [21:03:40] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3063 [21:03:42] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3496 [21:03:52] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3126 [21:03:52] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3468 [21:11:41] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1757.000000 [21:11:51] MySQL slave on rosemary is OK: Uptime: 8132763 Threads: 10 Questions: 2433345233 Slow queries: 1013712 Opens: 12121 Flush tables: 1 Open tables: 1019 Queries per second avg: 299.202 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1741 [21:12:40] MySQL slave on thyme is OK: Uptime: 3084137 Threads: 10 Questions: 1059321443 Slow queries: 521434 Opens: 151509 Flush tables: 1 Open tables: 2797 Queries per second avg: 343.474 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1743 [21:13:01] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3557 [21:13:21] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1578.000000 [21:17:50] MySQL slave on z-dat-s7-a is OK: Uptime: 211509 Threads: 24 Questions: 20477002 Slow queries: 8916 Opens: 123347 Flush tables: 1 Open tables: 3247 Queries per second avg: 96.813 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1610 [21:21:10] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1684.000000 [21:21:20] MySQL slave on daphne is OK: Uptime: 6560630 Threads: 51 Questions: 5409569671 Slow queries: 862984 Opens: 35401040 Flush tables: 3 Open tables: 16365 Queries per second avg: 824.550 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1651 [21:23:10] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [21:23:11] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12231.000000 [21:23:20] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [21:23:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:24:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42372 MB (4% inode=99%): [21:24:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:24:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:24:22] 3(commented) [ACCAPP-451] Urgent Account Creation <10https://jira.toolserver.org/browse/ACCAPP-451> (Cyberpower678) [21:27:21] s5 replag on daphne is CRITICAL: (Service Check Timed Out) [21:27:41] MySQL slave on z-dat-s3-a is OK: Uptime: 87023 Threads: 31 Questions: 84354949 Slow queries: 4769 Opens: 265636 Flush tables: 1 Open tables: 16384 Queries per second avg: 969.340 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1606 [21:29:01] MySQL slave on z-dat-s6-a is OK: Uptime: 272559 Threads: 13 Questions: 24854335 Slow queries: 16553 Opens: 153960 Flush tables: 1 Open tables: 1830 Queries per second avg: 91.188 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1740 [21:29:50] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [21:34:31] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3463.000000 [21:36:41] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3471.000000 [21:36:53] re [21:40:30] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1577.000000 [21:40:40] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1800.000000 [21:57:51] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.893555/1.00, alarm hl:np_load_long=0.851562/1.50, alarm hl:mem_free=22178.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.893555/1.10, alarm hl:np_load_long=0.851562/1.75, alarm hl:mem_free=22178.000000M/300M [21:58:51] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [22:08:41] @replag [22:08:42] Krinkle: s1-sec: 30s [+0.01 s/s]; s2/s5-pri-c: 2h 19m 24s [-1.74 s/s]; s3-rr: 1m 6s [-0.27 s/s]; s3-user: 1m 6s [-0.27 s/s]; s4-rr: 2h 19m 24s [-1.74 s/s]; s4-user: error [22:23:21] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [22:23:21] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8853.000000 [22:23:21] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [22:23:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:24:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42219 MB (4% inode=99%): [22:24:21] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:24:21] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:15:00] Free Memory on damiana.esi is WARNING: WARNING - 9.3% (389336 kB) free! [23:16:30] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.638672/1.75, alarm hl:np_load_avg=0.647949/2.00, alarm hl:mem_free=254.000000M/300M [23:19:01] Free Memory on damiana.esi is OK: OK - 10.2% (425924 kB) free. [23:19:31] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:23:01] Free Memory on damiana.esi is WARNING: WARNING - 9.6% (401012 kB) free! [23:23:20] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:23:21] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5126.000000 [23:23:21] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:23:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:24:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42055 MB (4% inode=99%): [23:24:21] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:24:21] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:27:31] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.704102/1.75, alarm hl:np_load_avg=0.739258/2.00, alarm hl:mem_free=269.000000M/300M [23:29:51] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [23:47:21] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3586.000000 [23:57:21] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1736.000000