[00:18:53] Free Memory on damiana.esi is WARNING: WARNING - 18.6% (777284 kB) free! [00:19:52] Free Memory on damiana.esi is OK: OK - 21.6% (904400 kB) free. [00:20:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [00:21:13] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:21:42] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3132 [00:22:23] Cluster on damiana.esi is CRITICAL: turnera FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote turnera Offline, check nfs-hasp, [00:22:53] Free Memory on damiana.esi is WARNING: WARNING - 17.9% (749320 kB) free! [00:22:53] Free Memory on turnera.esi is OK: OK - 88.4% (3700020 kB) free. [00:23:23] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 waiting, check nfs-hasp, [00:25:22] Cluster on damiana.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, nfs OFFLINE, check nfs-hasp, nfs-lh OFFLINE, nfs-home OFFLINE, [00:25:22] Cluster on turnera.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, nfs OFFLINE, check nfs-hasp, nfs-lh OFFLINE, nfs-home OFFLINE, [00:25:38] nacht ts [00:25:43] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.001693 secs [00:27:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:27:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43810 MB (4% inode=99%): [00:27:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:28:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:28:23] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [00:28:23] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [00:33:43] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3616 [00:36:23] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [00:38:43] NTP on turnera.esi is OK: NTP OK: Offset 0.003717 secs [00:48:23] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1963.000000 [00:57:53] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1817 [00:58:13] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1803.000000 [00:58:53] MySQL slave on rosemary is OK: Uptime: 7800784 Threads: 59 Questions: 2349241235 Slow queries: 991854 Opens: 11661 Flush tables: 1 Open tables: 920 Queries per second avg: 301.154 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1790 [00:59:13] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1784.000000 [01:02:54] Free Memory on damiana.esi is WARNING: WARNING - 17.2% (720516 kB) free! [01:16:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3643.000000 [01:18:18] @replag [01:18:23] Joan: s1-sec: 19m 29s [-]; s1-sec-c: 1h 2m 44s [-]; s3-rr: 4h 2m 33s [-]; s3-user: 4h 2m 33s [-]; s4-rr: 1h 2m 44s [-]; s5-rr: 10s [-]; s5-user: 10s [-]; s6-rr: 19m 44s [-] [01:18:23] Joan: s6-user: 19m 47s [-]; s7-rr: 1h 29m 19s [-]; s7-user: 1h 29m 19s [-] [01:27:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:27:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44581 MB (4% inode=99%): [01:27:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:27:51] Free Memory on damiana.esi is CRITICAL: CRITICAL - 6.1% (255652 kB) free! [01:28:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:28:30] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [01:31:51] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1840 [01:33:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6157 [01:34:21] 3(commented) [ET-45] February newsletter delivery <10https://jira.toolserver.org/browse/ET-45> (Keith Dorey) [01:34:51] Free Memory on damiana.esi is WARNING: WARNING - 10.5% (439956 kB) free! [01:37:51] Free Memory on damiana.esi is CRITICAL: CRITICAL - 10.0% (420500 kB) free! [01:42:21] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1825.000000 [01:42:51] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1817 [01:43:21] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1794.000000 [01:46:21] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1846.000000 [01:47:31] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.561035/1.75, alarm hl:np_load_avg=0.540039/2.00, alarm hl:mem_free=272.000000M/300M [01:51:31] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:57:31] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.721191/1.75, alarm hl:np_load_avg=0.727051/2.00, alarm hl:mem_free=299.000000M/300M [02:01:51] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.0% (165960 kB) free! [02:08:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3648 [02:17:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7246.000000 [02:20:23] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1776.000000 [02:21:03] MySQL slave on rosemary is OK: Uptime: 7805714 Threads: 62 Questions: 2350188449 Slow queries: 992772 Opens: 11670 Flush tables: 1 Open tables: 919 Queries per second avg: 301.85 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1736 [02:27:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:28:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44434 MB (4% inode=99%): [02:28:03] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:28:33] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [02:28:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:34:04] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9290 [02:35:23] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1878.000000 [02:36:04] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1900 [03:01:01] @replag [03:01:07] juancarlos: s1-sec: 36m 17s [+0.16 s/s]; s1-sec-c: 2h 29m 44s [+0.85 s/s]; s2/s5-pri-c: 13s [-]; s3-rr: 5h 45m 16s [+1.00 s/s]; s3-user: 5h 45m 20s [+1.00 s/s]; s4-rr: 2h 29m 31s [+0.84 s/s]; s6-rr: 1h 37m 13s [+0.75 s/s]; s6-user: 1h 37m 13s [+0.75 s/s] [03:01:08] juancarlos: s7-rr: 2h 53m 44s [+0.82 s/s]; s7-user: 2h 53m 44s [+0.82 s/s] [03:02:04] Free Memory on damiana.esi is CRITICAL: CRITICAL - 3.2% (133688 kB) free! [03:02:33] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.019043/1.75, alarm hl:np_load_avg=1.040527/2.00, alarm hl:mem_free=295.000000M/300M [03:03:35] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:07:33] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1787.000000 [03:07:33] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.930664/1.75, alarm hl:np_load_avg=0.987793/2.00, alarm hl:mem_free=221.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.930664/1.50, alarm hl:np_load_long=0.989258/1.75, alarm hl:mem_free=221.000000M/250M [03:08:04] MySQL slave on rosemary is OK: Uptime: 7808535 Threads: 54 Questions: 2350745224 Slow queries: 993467 Opens: 11673 Flush tables: 1 Open tables: 920 Queries per second avg: 301.48 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1755 [03:08:04] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6070 [03:17:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9240.000000 [03:27:34] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:28:33] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [03:28:34] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:29:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44284 MB (4% inode=99%): [03:29:04] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:34:03] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11684 [03:40:04] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1884 [03:40:33] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1902.000000 [03:42:04] MySQL slave on rosemary is OK: Uptime: 7810576 Threads: 56 Questions: 2351146283 Slow queries: 993865 Opens: 11675 Flush tables: 1 Open tables: 920 Queries per second avg: 301.20 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1794 [03:42:33] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1722.000000 [03:47:10] Has anyone noticed simple queries randomly timing out? I recently had this query killed: "SELECT user_editcount FROM user WHERE user_name = 'Some Username'". [03:47:49] Is it possible some other query is locking or something? [03:48:07] I had a suspicion that queries with subqueries were causing weird query build-ups at some point. [03:48:12] MySQL is evil. [03:49:45] Possibly; I'm not sure how it's set up serverside. [03:51:38] I've been getting the occasional query-killed notification for a while, but tonight I'm getting flooded with notifications for trivial queries. My scripts are getting heavier load than usual due to the steward confirmation links, so maybe there's some sort of load or connection limit that causes queries to simply stall? [04:02:04] Free Memory on damiana.esi is CRITICAL: CRITICAL - 3.9% (164756 kB) free! [04:02:34] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.581543/1.75, alarm hl:np_load_avg=1.561035/2.00, alarm hl:mem_free=255.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.581543/1.50, alarm hl:np_load_long=1.312012/1.75, alarm hl:mem_free=255.000000M/250M [04:07:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:09:03] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6663 [04:17:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10633.000000 [04:17:34] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.148438/1.75, alarm hl:np_load_avg=1.195801/2.00, alarm hl:mem_free=250.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.148438/1.50, alarm hl:np_load_long=1.239746/1.75, alarm hl:mem_free=250.000000M/250M [04:27:34] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:28:33] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [04:29:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44140 MB (4% inode=99%): [04:29:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:30:05] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:34:04] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14330 [04:48:29] ...and the toolserver is down. [04:56:18] Pathoschild: but you are stil existing, i an not according to toolserver: merl@willow:~$ become merliwbot [04:56:18] sudo: unknown uid 1346: who are you? [04:59:18] Hmm. [05:15:17] man, this does not look good [05:15:24] toolserver in flames? [05:15:38] @replag [05:15:39] Yep. [05:15:55] :'-( [05:16:53] been flaky for the last few hours [05:17:05] but now it is gone (ssh and http) [05:19:33] somebody should update status.toolserver.org [05:25:02] so..... nobody knows anything about what's going on with the tollserver right now? [05:25:33] time to go to sleep then and hope for the best tomorrow. [05:27:24] OK, so I've got a response. But now it's just a 404 where I know there's a page. [05:31:11] NTP on rosemary is OK: NTP OK: Offset 0.000328 secs [05:31:11] SMF on rosemary is OK: OK - all services online [05:31:11] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2029.000000 [05:31:20] SMTP on rosemary is OK: SMTP OK - 0.032 sec. response time [05:31:21] FMA on rosemary is OK: OK [05:31:21] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2718.000000 [05:31:30] Load avg. on rosemary is OK: OK - load average: 0.22, 0.16, 0.25 [05:31:30] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2724.000000 [05:31:40] MySQL on rosemary is OK: Uptime: 7817153 Threads: 16 Questions: 2352016378 Slow queries: 994534 Opens: 11677 Flush tables: 1 Open tables: 920 Queries per second avg: 300.878 [05:31:40] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13625.000000 [05:31:40] Cluster on turnera.esi is CRITICAL: damiana FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote damiana Offline, check nfs-hasp, check apache, check tsbot, check squid, check sge-qmaster, check nagios, check named, check haproxy-sql, [05:31:40] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [05:31:49] MySQL slave on rosemary is OK: Uptime: 7817160 Threads: 18 Questions: 2352016403 Slow queries: 994534 Opens: 11677 Flush tables: 1 Open tables: 920 Queries per second avg: 300.878 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [05:31:50] MySQL slave on z-dat-s7-a is OK: Uptime: 139413 Threads: 3 Questions: 12462212 Slow queries: 13812 Opens: 3691 Flush tables: 1 Open tables: 3388 Queries per second avg: 89.390 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [05:31:59] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [05:32:00] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [05:32:00] PING on rosemary is OK: PING OK - Packet loss = 0%, RTA = 0.18 ms [05:32:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:32:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44125 MB (4% inode=99%): [05:32:10] RAID on rosemary is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:33:59] Free Memory on damiana.esi is OK: OK - 77.1% (3228368 kB) free. [05:34:11] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2644.000000 [05:34:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:34:41] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 waiting, check nfs-hasp, [05:35:40] DiskSuite on damiana.esi is CRITICAL: CRITICAL - submirror d11 of mirror d10 is Resyncing and submirror d12 of mirror d10 is Resyncing [05:36:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:36:31] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [05:36:41] DiskSuite on damiana.esi is OK: OK - No disk failures detected [05:46:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [05:46:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [05:50:11] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [05:58:12] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3648.000000 [06:00:31] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [06:00:40] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [06:07:20] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3631.000000 [06:07:31] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3640.000000 [06:24:31] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4480 [06:26:11] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3329.000000 [06:26:31] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3094 [06:29:12] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1492.000000 [06:29:20] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 2.000000 [06:29:32] MySQL slave on daphne is OK: Uptime: 6247915 Threads: 14 Questions: 5222596548 Slow queries: 786051 Opens: 33973718 Flush tables: 3 Open tables: 16108 Queries per second avg: 835.894 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1210 [06:30:51] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3020 [06:31:41] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [06:31:41] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17228.000000 [06:31:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5839 [06:31:51] MySQL slave on z-dat-s7-a is OK: Uptime: 143015 Threads: 5 Questions: 13140257 Slow queries: 13903 Opens: 3776 Flush tables: 1 Open tables: 3387 Queries per second avg: 91.880 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1692 [06:32:01] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [06:32:01] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [06:32:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:32:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43953 MB (4% inode=99%): [06:32:51] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6023 [06:34:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:36:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:42:30] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3426.000000 [06:42:42] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3398 [06:45:31] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3449.000000 [06:45:50] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3388 [06:54:30] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1585.000000 [06:54:41] MySQL slave on thyme is OK: Uptime: 2773455 Threads: 11 Questions: 958153690 Slow queries: 478250 Opens: 139247 Flush tables: 1 Open tables: 2825 Queries per second avg: 345.472 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1545 [06:57:31] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1656.000000 [06:57:50] MySQL slave on rosemary is OK: Uptime: 7822323 Threads: 14 Questions: 2352655694 Slow queries: 994936 Opens: 11679 Flush tables: 1 Open tables: 920 Queries per second avg: 300.761 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1566 [06:58:11] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7249.000000 [07:08:30] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7302.000000 [07:18:08] zzz [07:31:42] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [07:31:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7122.000000 [07:32:12] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [07:32:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43812 MB (4% inode=99%): [07:32:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:33:00] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [07:34:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:35:44] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3121.000000 [07:36:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:36:42] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3055.000000 [07:36:42] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1777.000000 [07:38:42] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1278.000000 [07:45:13] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3095.000000 [07:50:13] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1424.000000 [08:20:43] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 17634 MB (1% inode=99%): [08:31:43] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [08:32:13] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [08:32:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44620 MB (4% inode=99%): [08:32:14] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:33:13] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [08:34:44] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:36:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:15:57] @replag all [09:15:58] nosy: s1-pri: 0s [-]; s1-sec: 0s [-]; s1-sec-c: 0s [-]; s2-pri: 3s [-]; s2/s5-pri-c: 4s [-]; s3-rr: 12h 12s [-]; s3-user: 12h 12s [-]; s4-rr: 1s [-] [09:16:00] nosy: s4-user: 4s [-]; s5-rr: 5s [-]; s5-user: 6s [-]; s6-rr: error; s6-user: error; s7-rr: 3s [-]; s7-user: 3s [-] [09:25:42] nosy: do you plan to restart nightshade during the maintenance? [09:25:59] Danny_B|backup: no [09:26:34] great. please don'ť forget to update the wiki page, so we can know what has been done, thank you [09:30:15] @replag all [09:30:17] nosy: s1-pri: 0s [-]; s1-sec: 0s [-]; s1-sec-c: 2s [+0.00 s/s]; s2-pri: 5s [+0.00 s/s]; s2/s5-pri-c: 8s [+0.00 s/s]; s3-rr: 10h 56m 20s [-4.47 s/s]; s3-user: 10h 56m 20s [-4.47 s/s]; s4-rr: 3s [+0.00 s/s] [09:30:17] nosy: s4-user: 8s [+0.00 s/s]; s5-rr: 2s [-0.00 s/s]; s5-user: 12s [+0.01 s/s]; s6-rr: error; s6-user: error; s7-rr: 7s [+0.00 s/s]; s7-user: 7s [+0.00 s/s] [09:48:30] @replag all [09:48:30] nosy: s1-pri: 0s [-]; s1-sec: 0s [-]; s1-sec-c: 1s [-0.00 s/s]; s2-pri: 7s [+0.00 s/s]; s2/s5-pri-c: 3m 45s [+0.20 s/s]; s3-rr: 7h 52m 17s [-10.09 s/s]; s3-user: 7h 52m 17s [-10.09 s/s]; s4-rr: 1s [-0.00 s/s] [09:48:31] nosy: s4-user: 3m 45s [+0.20 s/s]; s5-rr: 1s [-0.00 s/s]; s5-user: 5s [-0.01 s/s]; s6-rr: 3h 56m 9s [-]; s6-user: 3h 56m 9s [-]; s7-rr: 3s [-0.00 s/s]; s7-user: 3s [-0.00 s/s] [10:02:54] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.497070/1.75, alarm hl:np_load_avg=0.500000/2.00, alarm hl:mem_free=250.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.497070/1.50, alarm hl:np_load_long=0.452149/1.75, alarm hl:mem_free=250.000000M/250M [10:05:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [10:10:23] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3079 [10:13:23] MySQL slave on z-dat-s6-a is OK: Uptime: 2808 Threads: 6 Questions: 748361 Slow queries: 91 Opens: 278 Flush tables: 1 Open tables: 262 Queries per second avg: 266.510 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1559 [10:16:53] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3267 [10:19:52] MySQL slave on z-dat-s3-a is OK: Uptime: 156692 Threads: 10 Questions: 120997739 Slow queries: 19358 Opens: 517245 Flush tables: 1 Open tables: 16384 Queries per second avg: 772.201 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1072 [10:33:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:33:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44371 MB (4% inode=99%): [10:35:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:36:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:17:21] 3(created) [ACCAPP-449] Account Approval for the toolserver withing the RENDER project; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-449> (Gerrit Holz) [11:33:34] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44243 MB (4% inode=99%): [11:33:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:35:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:36:34] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:58:04] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.466797/1.75, alarm hl:np_load_avg=0.473633/2.00, alarm hl:mem_free=287.000000M/300M [12:03:23] 3(created) [MAGNUS-293] Agencia Brazil search in FIST is broken; Magnus' tools: FIST; Bug <10https://jira.toolserver.org/browse/MAGNUS-293> (bulwersator) [12:05:05] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:14:04] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.469238/1.75, alarm hl:np_load_avg=0.497070/2.00, alarm hl:mem_free=271.000000M/300M [12:33:34] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44103 MB (4% inode=99%): [12:34:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:35:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:36:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:54:34] why is my quota suddenly 0 insetad of 256M? I asked to have it raised to 1024M [12:55:21] now I tried to edit one of my files in Emacs, and at save, it was trunkated to 0 bytes. [13:04:20] Hello all [13:05:27] ' [13:06:02] Cluster on damiana.esi is CRITICAL: check nfs-hasp, nfs-home Online, [13:06:12] Cluster on turnera.esi is CRITICAL: check nfs-hasp, nfs-home Online, [13:07:02] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [13:07:12] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [13:08:40] DaBPunkt: I find myself having 0 quota, my files get trunkated on wolfsbane [13:09:12] quite the opposite of https://jira.toolserver.org/browse/TS-1266 [13:09:28] lt me look [13:11:13] this is a disaster, my files generated by my cron script are 0 bytes [13:12:32] several weeks of data collection is gone [13:13:35] LA2: I increased the quota nearly 1 wweek ago, how can "several weeks" are gone than? [13:15:04] [[User:LA2]] ! 10https://wiki.toolserver.org/w/index.php?diff=6642&oldid=6426&rcid=8755 * LA2 * (+212) () [13:15:37] what is my quota right now? [13:15:48] quota -v says "0" [13:16:31] It should be 1GB soft, 1.5GB hard [13:17:05] when I tried to edit a file, it got trunkated to 0 bytes, with an error message that my quota was exceeded. [13:17:31] I see files in your home with are 0 byes from decembre [13:17:46] I will re-set your quota [13:18:58] I was above my 256 soft limit, but I could still create files. I don't know the time limit between soft and hard. [13:20:13] ok, I found the problem with the quotadefination [13:20:22] now I see my increased quota, thanks! [13:22:09] [[Admin:Quota]] 10https://wiki.toolserver.org/w/index.php?diff=6643&oldid=6353&rcid=8756 * Dab * (+179) () [13:22:38] LA2: I'm sorry that some data was lost because of me [13:23:19] okay [13:23:39] i was late in applying for the increased quota as well [13:23:51] but why, why, why do we have to have a quota system? [13:23:59] this is 2012, not 1982 [13:24:15] we're the world's 6th most used website [13:24:51] LA2: most part, because users otherwise create HUGE log-files by mistake [13:25:08] I deleted several >1GB or more [13:25:12] so allow them to do this, up to the level they pay for [13:25:30] just tell me what the price for 2 GB, 20 GB or 200 GB is [13:25:54] but don't apply a random quota of 256 MB [13:26:27] LA2: if you need more space, just tell me. We have 88GB left, you have have it (or most parts of it) ;) [13:26:43] I want 200 GB, how much does that cost? [13:27:07] we have only 88 [13:27:13] even if you give me 200 GB quota, and the disk has 88 GB, that doesn't give me 200 GB [13:27:44] tomorrow the disk could fill up, and I would still not be able to save my files. I don't want a quota, I want disk space. [13:28:37] 88 GB is nothing [13:28:42] tomorrow the disk could fill up – no, because we HAVE quotas, that can not happen [13:29:08] so you can't give me a quota of 88 GB then [13:29:37] do you keep count of the total quotas? What if all users would fill their current quota, how much disk space would be left? [13:29:48] no, it was a joke of course. But I can give you more space if you need it, that was true [13:30:09] I don't ever want to see 0 byte files again. How much would that cost? [13:31:35] I can't believe I'm having this conversation in 2012. [13:31:50] WMDE has so much money. Go buy some disk. How hard can that be? [13:32:48] trust me: very hard. We still wait for a delivery which should have arrived in teh datacenter in the 3 calendar week [13:32:52] [[User:LA2]] !M 10https://wiki.toolserver.org/w/index.php?diff=6644&oldid=6642&rcid=8757 * LA2 * (+14) () [13:34:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43955 MB (4% inode=99%): [13:34:37] LA2: But if you like, you can write to the WMDE and tell them, that the TS needs more money – then I would be not the only one who does [13:34:44] LA2: do you know that the currently global produced hard disk space is only about 40% than that was produced one year ago? [13:35:06] Merlissimo: stop trolling, this is out of topic [13:35:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:35:28] :-P [13:35:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:37:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:40:42] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:41:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:55:42] MySQL slave on z-dat-s4-a is WARNING: No slaves defined [13:56:31] MySQL on z-dat-s4-a is OK: Uptime: 62 Threads: 3 Questions: 55 Slow queries: 0 Opens: 0 Flush tables: 1 Open tables: 0 Queries per second avg: 0.887 [13:57:42] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:59:31] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:03:46] DaBPunkt: db on s4 will be off for a long time, or is expected to be fixed soon? [14:04:12] Alchimista: tonight or in the morning, depens how long the import takes [14:04:39] DaBPunkt: ok. one tool for the stews election was using it, so we are in the darkness :P [14:10:02] Lars Aronsson * [Toolserver-l] Toolserver quota [14:23:23] Daniel_WMDE: do you have a moment? [14:23:43] sure [14:23:46] wassup? [14:23:59] Daniel_WMDE: did you already poll this room re the date for WMDE hackathon? [14:26:03] no. but I'm pretty much comitted to the June date by now. [14:26:08] you could poll toolserver-l if you want. [14:26:24] but i doubt that there's be a conclusive answer, the community ios large and diverse [14:27:44] Daniel_WMDE: oh, are we? ok. [14:27:50] * sumanah stops doing polling work [14:29:02] DaB. * Re: [Toolserver-l] Toolserver quota [14:30:13] sumanah: well, i#m basically doing the final round of polling inside wmde. your input helped a lot, and i was going to write to you next :) [14:30:22] Oh! Glad it helped [14:32:15] sumanah: i'm about to send a mail to pavel to finalize the date. we can go on polling, but i guess we should simply set the date rather sooner than later. [14:33:03] Daniel_WMDE: YES. Um. I feel bad because I told people "please reply within the next 2 days" when I emailed last night. But you can blame me for that delayed communication if they get mad [14:33:10] I should have sent all of that on Monday [14:33:36] but it took a few uninterrupted hours to translate and read something like 25 chapters' plans/sites/project pages [14:33:41] sumanah: don't feel bad. i also have to collect final feedback [14:33:52] but i don't see much point in asking more people now [14:33:58] unless their presence is crucial. [14:34:07] Yeah. And Tim can only make June. [14:34:17] which basically settles it. i need him there. [14:34:21] I thought it might. [14:34:24] and angelica can make it work [14:34:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43807 MB (4% inode=99%): [14:35:02] (hm, i shouldn't change the spelling of names just because i'm thinking in english...) [14:35:21] Ha! [14:35:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:35:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:37:20] 3(created) [ACCAPP-450] Reactivation of the account aadam; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-450> (Angelika Adam) [14:37:20] A reminder to Toolserver folks: You can apply for a Wikimania travel subsidy: https://wikimania2012.wikimedia.org/wiki/Scholarships and the deadline is 16 February. [14:37:32] MySQL on z-dat-s4-a is OK: Uptime: 7 Threads: 1 Questions: 3 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 8 Queries per second avg: 0.428 [14:37:42] MySQL slave on z-dat-s4-a is WARNING: No slaves defined [14:37:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:38:43] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:40:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:42:22] sumanah: probably better to write to toolserver-l, this channel is pretty dead. [14:42:36] ok, good to know Daniel_WMDE [14:43:24] [[~arnomane/WP-autoreviewer]] ! 10https://wiki.toolserver.org/w/index.php?diff=6645&oldid=4090&rcid=8758 * 121.247.78.194 * (+3) (you could have used the tool on THIS PAGE) [14:52:02] Sumana Harihareswara * [Toolserver-l] Apply for Wikimania travel funding? Deadline in 8 days [15:05:20] 3(moved) [TS-1280] Reactivation of the account aadam <10https://jira.toolserver.org/browse/TS-1280> (DaB.) [15:10:21] 3(resolved) [TS-1280] Reactivation of the account aadam <10https://jira.toolserver.org/browse/TS-1280> (DaB.) [15:14:20] 3(commented) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [15:19:02] Platonides * Re: [Toolserver-l] Toolserver quota [15:32:13] toolserver kicked my ssh session [15:32:25] seems to be down again. are they still doing maintenance? [15:32:32] no [15:32:48] works for me [15:34:13] DaBPunkt: is the error message at http://toolserver.org/~pathoschild/accounteligibility/?user=Lord_Roem&wiki=&event=23 related to https://jira.toolserver.org/browse/TS-1277 because it's on s4? [15:34:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43634 MB (4% inode=99%): [15:35:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:35:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:36:24] tsnag: yes, but the tool-author could fix it [15:36:51] Thehelpfulone: yes, but the tool-author could fix it [15:37:17] okay, do they switch to anther s* ? [15:37:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:38:42] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [15:39:05] Thehelpfulone: he/she would have to use sql-s4-rr.toolserver.org instead of the very old sql-s4.toolserver.org [15:39:11] have → has [15:39:16] okay thanks [15:40:32] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [16:00:46] how to move a jira ticket to another project? [16:00:48] it's https://jira.toolserver.org/browse/DBQ-173 [16:01:22] liangent: is there a "Move" menu-point under "More Actions" for you? [16:01:54] DaBPunkt: no [16:02:07] liangent: ok. to where you I move it? [16:02:13] +shouldf [16:02:21] -you [16:02:44] DaBPunkt: I think TS [16:02:58] ah ok [16:05:25] 3(moved) [TS-1281] problem with fr.wiki data base <10https://jira.toolserver.org/browse/TS-1281> (DaB.) [16:34:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44368 MB (4% inode=99%): [16:35:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:35:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:37:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:38:42] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [16:40:32] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [17:31:52] Free Memory on turnera.esi is WARNING: WARNING - 19.7% (824400 kB) free! [17:34:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44183 MB (4% inode=99%): [17:36:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:36:03] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:38:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:38:52] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [17:40:31] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [17:43:25] @replag [17:43:26] Quentinv57: s1-sec: 7m 21s [+0.03 s/s]; s1-sec-c: 32s [+0.00 s/s]; s2-pri: 13s [+0.00 s/s]; s3-rr: 24m 40s [+0.11 s/s]; s3-user: 24m 40s [+0.11 s/s]; s4-rr: 32s [+0.00 s/s]; s4-user: error; s6-rr: 15m 59s [+0.07 s/s] [17:43:27] Quentinv57: s6-user: 15m 59s [+0.07 s/s]; s7-rr: 18m 17s [+0.08 s/s]; s7-user: 18m 17s [+0.08 s/s] [17:47:52] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [17:48:52] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [17:52:12] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1964 [17:55:52] MySQL slave on z-dat-s6-a is OK: Uptime: 571 Threads: 1 Questions: 13 Slow queries: 0 Opens: 23 Flush tables: 1 Open tables: 16 Queries per second avg: 0.22 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1675 [17:56:51] MySQL on z-dat-s6-a is OK: Uptime: 625 Threads: 3 Questions: 9665 Slow queries: 1 Opens: 146 Flush tables: 1 Open tables: 137 Queries per second avg: 15.464 [18:10:35] MySQL slave on z-dat-s3-a is OK: Uptime: 184936 Threads: 26 Questions: 141789137 Slow queries: 21922 Opens: 586746 Flush tables: 1 Open tables: 16384 Queries per second avg: 766.693 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1752 [18:31:54] Free Memory on turnera.esi is WARNING: WARNING - 14.9% (622964 kB) free! [19:11:24] 3(resolved) [MAGNUS-293] Agencia Brazil search in FIST is broken <10https://jira.toolserver.org/browse/MAGNUS-293> (Magnus Manske) [19:27:02] re [19:27:05] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.155274/1.00, alarm hl:np_load_long=0.664062/1.50, alarm hl:mem_free=22476.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.155274/1.10, alarm hl:np_load_long=0.664062/1.75, alarm hl:mem_free=22476.000000M/300M [19:27:26] Maintenance is staring now [19:27:29] +t [19:28:05] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [19:31:55] Free Memory on turnera.esi is WARNING: WARNING - 13.6% (569268 kB) free! [19:34:22] 3(commented) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [19:34:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43723 MB (4% inode=99%): [19:35:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:36:06] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:38:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:38:21] 3(commented) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [19:38:56] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [19:40:45] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [19:45:23] 3(commented) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [19:58:55] Free Memory on turnera.esi is CRITICAL: CRITICAL - 9.9% (416188 kB) free! [19:59:39] hello DaBPunkt . There seems to be some issues with the database, could you please take a look at it ? [19:59:51] Quentinv57: what is the problem? [20:00:24] DaBPunkt: for example my SULinfo tool takes very long to load the last days and I get lots of mail saying that my query got interrupted [20:00:45] but I don't want to add /* SLOW OK */ because that should not be used for tools in my opinion, and not for such query [20:00:55] Free Memory on turnera.esi is WARNING: WARNING - 12.4% (517464 kB) free! [20:01:05] before, the SULinfo page took less than one minute to load completely, so that's not normal [20:01:19] Quentinv57: ask me tomorrow again. I do maintenance at the moment [20:01:32] but it could just fix itself when sql-s4 is back [20:01:51] DaBPunkt: when s4 will be back ? [20:02:02] tomorrow morning [20:02:04] yes, that's the problem : Could not connect to database host 'sql-s4.toolserver.org' [20:02:17] Quentinv57: do you write to sql-s4? [20:02:56] DaBPunkt: not at all, I don't know what Pathoschild's tool is doing (because we would need it to check votes at steward election) [20:03:23] my tools do not write on TS databases personally [20:03:50] Quentinv57: than use sql-s4-rr.toolserver.org instead [20:05:46] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1929.000000 [20:08:31] DaBPunkt: okay thanks [20:08:41] np [20:20:45] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 18953 MB (1% inode=99%): [20:21:38] toolserver: 03dab * r1132 10/trunk/ts-specs/base-specs/librsvg.spec: -Updated librsvg to 2.35.2. [20:24:20] 3(resolved) [TS-1274] Update rsvg to 2.35.1 <10https://jira.toolserver.org/browse/TS-1274> (DaB.) [20:33:46] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [20:35:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43534 MB (4% inode=99%): [20:35:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:36:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:38:16] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:39:15] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [20:39:16] Free Memory on turnera.esi is CRITICAL: CRITICAL - 7.9% (331148 kB) free! [20:40:15] Free Memory on turnera.esi is WARNING: WARNING - 16.4% (687916 kB) free! [20:40:45] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [20:50:15] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.2% (343424 kB) free! [20:51:11] DaBPunkt around? [20:51:39] ToAruShiroiNeko: yes, but we are in maintenance [20:52:07] ToAruShiroiNeko: if it is for your query: You can ask ANY toolserver-user to run it BTW :) [20:52:23] yes but I only trust the DaBPunkt :) [20:52:25] :p [20:55:10] ToAruShiroiNeko: when sql-s4 is back tomorrow, I will run your query [21:10:31] ok [21:10:33] thank you [21:12:12] s4 is season 4? [21:12:14] :p [21:13:08] ToAruShiroiNeko: sun-time 4. 1 sun-time is the time a normal sun needs until it explodes (~10 Bilion years) ;) [21:13:21] no [21:13:31] sun will blow up in 5 billion [21:13:42] we will all get sun tanned to the max [21:14:00] ToAruShiroiNeko: yes, but I burned already for ~5 billion years [21:14:18] ah [21:14:19] yes [21:14:25] not really burned tho [21:14:36] no cumbustion involved [21:21:25] 3(commented) [TS-1274] Update rsvg to 2.35.1 <10https://jira.toolserver.org/browse/TS-1274> (Jarry1250) [21:33:16] Free Memory on turnera.esi is CRITICAL: CRITICAL - 9.4% (393428 kB) free! [21:34:46] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7272.000000 [21:35:56] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:36:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44306 MB (4% inode=99%): [21:36:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:38:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:39:15] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [21:40:45] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [21:55:08] Hi [21:55:21] hello erwin [21:55:54] My tools can't connect to MySQL at the moment (Access denied). Did anything change? Should I do something? [22:00:10] (I can connect from the CLI) [22:02:15] Free Memory on turnera.esi is WARNING: WARNING - 15.6% (651128 kB) free! [22:04:30] DaBPunkt, do you have any idea perhaps? [22:04:56] erwin: which server do you try? [22:05:52] See the link I pm'ed. I tried just using mysql / sql on wolfsbane and nightshade and that works fine. [22:08:17] erwin: to whcih server to you like to connect? [22:09:20] 3(created) [DBQ-176] We need a shared account for me together with Anselm Metzger from Wikimedia Deutschland.; Database Queries; Task <10https://jira.toolserver.org/browse/DBQ-176> (Philipp Zedler) [22:09:27] I just use sql, sql-s1, sql-s2 etc. In this case it's sql-s2. [22:12:34] erwin: there is only a known problem with sql-s4 at the moment [22:16:03] Ah.. that seems to be causing it. I don't parse the status files (/var/www/status_s4). The first line not starting with a # is the current status, right? [22:18:38] yes. The problem is that only sql-s4-user is down, not sql-s4-rr and our status-files can not show that at the moment [22:19:17] OK, thanks for the help. :) [22:25:03] *Don't parse the files correctly. But I guess in this case that wouldn't have made a difference. [22:25:06] GTG Bye [22:33:24] 3(moved) [TS-1282] We need a shared account for me together with Anselm Metzger from Wikimedia Deutschland. <10https://jira.toolserver.org/browse/TS-1282> (DaB.) [22:34:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10873.000000 [22:35:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:36:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44158 MB (4% inode=99%): [22:36:25] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:38:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:39:15] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [22:40:45] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [23:00:25] Free Memory on turnera.esi is CRITICAL: CRITICAL - 3.4% (143936 kB) free! [23:01:25] Free Memory on turnera.esi is WARNING: WARNING - 15.5% (647960 kB) free! [23:35:46] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14532.000000 [23:36:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43987 MB (4% inode=99%): [23:36:25] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:36:54] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:38:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:39:15] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [23:40:44] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [23:48:32] [[Special:Log/newusers]] create 10 * JonalynAcero00 * (New user account) [23:50:03] [[User talk:JonalynAcero00]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6646&rcid=8760 * JonalynAcero00 * (+3539) (Created page with "Ever wondered the reason why carpet cleaning seems to be the Huge job? Even if we make use of vacuum cleaners we do not fully get persuaded. The sparkle on the carpets and rugs a...") [23:50:42] [[Special:Log/delete]] delete 10 * Dab * (deleted "[[02User talk:JonalynAcero0010]]": SPAM) [23:51:12] [[Special:Log/block]] block 10 * Dab * (blocked [[02User:JonalynAcero0010]] with an expiry time of infinite (account creation disabled, e-mail blocked): SPAMMER) [23:56:27] nacht ts [23:57:02] DaB. * Re: [Toolserver-announce] [Toolserver-l] Regular Maintenance at 8. February