[00:02:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:05:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:09:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:31:40] [[Special:Log/newusers]] create 10 * Luckas Blade * (New user account) [00:35:16] nacht ts [00:35:36] [[Interwiki bot MMP planning]] ! 10https://wiki.toolserver.org/w/index.php?diff=6847&oldid=6834&rcid=9021 * Luckas Blade * (+235) () [00:36:21] [[User:Luckas Blade]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6848&rcid=9022 * Luckas Blade * (+24) (Created page with "[[:m:user:Luckas Blade]]") [00:41:40] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 401350 MB (7% inode=41%): [00:53:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31311 MB (3% inode=99%): [01:02:21] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:05:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:09:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:13:10] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.162109/1.10, alarm hl:np_load_long=0.863281/1.55, alarm hl:mem_free=19155.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.162109/1.00, alarm hl:np_load_long=0.863281/1.50, alarm hl:mem_free=19155.000000M/300M, alarm hl:available=1/0 [01:13:20] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.437988/1.8, alarm hl:np_load_avg=0.823242/2.3, alarm hl:mem_free=261.000000M/300M, alarm hl:available=1/0 [01:14:10] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [01:20:20] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:41:40] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 412938 MB (7% inode=41%): [01:43:11] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.090820/1.00, alarm hl:np_load_long=0.856445/1.50, alarm hl:mem_free=19032.000000M/300M, alarm hl:available=1/0 [01:45:10] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [02:03:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:06:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:10:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:41:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 408035 MB (7% inode=41%): [03:03:25] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:06:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:10:16] good night [03:10:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:41:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405399 MB (7% inode=41%): [03:55:25] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:55:35] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:44] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:55:45] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:55:45] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:55:45] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:55:56] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:56:15] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 2748 MB (99% inode=99%): [03:56:16] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 46592 MB (11% inode=99%): [03:56:16] SMF on z-dat-s4-a is OK: OK - all services online [03:56:16] / on z-dat-s4-a is OK: DISK OK - free space: / 11659 MB (38% inode=87%): [03:56:26] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:03:26] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:06:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:10:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:17:25] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.881836/1.8, alarm hl:np_load_avg=0.951660/2.3, alarm hl:mem_free=218.000000M/300M, alarm hl:available=1/0 [04:19:25] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:41:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 403181 MB (7% inode=41%): [04:44:15] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.216797/1.10, alarm hl:np_load_long=0.868164/1.55, alarm hl:mem_free=18558.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.216797/1.00, alarm hl:np_load_long=0.868164/1.50, alarm hl:mem_free=18558.000000M/300M, alarm hl:available=1/0 [04:45:15] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [05:03:25] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:06:36] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:10:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:12:25] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.596191/1.8, alarm hl:np_load_avg=1.274902/2.3, alarm hl:mem_free=159.000000M/300M, alarm hl:available=1/0: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.596191/1.9, alarm hl:np_load_long=1.089355/2.25, alarm hl:mem_free=159.000000M/200M, alarm hl:available=1/0 [05:15:16] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 15.508 second response time [05:23:15] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 13.600 second response time [05:24:25] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:42:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 399061 MB (7% inode=40%): [06:01:20] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.236 second response time [06:03:42] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:06:30] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 14.947 second response time [06:06:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:09:29] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 16.256 second response time [06:10:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:14:41] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.459961/1.8, alarm hl:np_load_avg=1.478516/2.3, alarm hl:mem_free=232.000000M/300M, alarm hl:available=1/0 [06:16:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:21:41] Load avg. on willow is WARNING: WARNING - load average: 17.04, 15.73, 13.25 [06:21:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=2.314941/1.8, alarm hl:np_load_avg=1.994629/2.3, alarm hl:mem_free=736.000000M/300M, alarm hl:available=1/0: longrun@willow exceedes load threshold: alarm hl:np_load_short=2.314941/1.9, alarm hl:np_load_long=1.661621/2.25, alarm hl:mem_free=736.000000M/200M, alarm hl:available=1/0 [06:22:41] Load avg. on willow is OK: OK - load average: 12.46, 14.69, 13.04 [06:32:43] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.910 second response time [06:33:30] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 16.718 second response time [06:42:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 397239 MB (7% inode=40%): [06:47:42] Load avg. on willow is WARNING: WARNING - load average: 21.27, 17.95, 14.68 [06:51:41] Load avg. on willow is OK: OK - load average: 9.82, 14.57, 14.10 [07:03:42] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.513 second response time [07:03:42] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:04:30] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 17.380 second response time [07:06:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:10:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:17:41] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:42:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461839 MB (8% inode=44%): [08:03:45] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:06:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:11:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:42:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461730 MB (8% inode=44%): [09:04:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:07:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:07:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:11:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:43:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461587 MB (8% inode=44%): [09:57:03] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.806152/1.8, alarm hl:np_load_avg=0.783691/2.3, alarm hl:mem_free=261.000000M/300M, alarm hl:available=1/0 [09:58:03] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [10:03:03] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.002441/1.8, alarm hl:np_load_avg=0.953125/2.3, alarm hl:mem_free=272.000000M/300M, alarm hl:available=1/0 [10:04:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:07:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:07:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:11:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:13:34] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:14:03] SMF on hyacinth is OK: OK - all services online [10:23:54] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.059570/1.00, alarm hl:np_load_long=0.869140/1.50, alarm hl:mem_free=18551.000000M/300M, alarm hl:available=1/0 [10:24:54] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [10:39:24] [[Special:Log/newusers]] create 10 * Vinod rakte * (New user account) [10:44:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461109 MB (8% inode=44%): [10:55:04] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:04] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:04] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:13] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:34] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:34] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:56:03] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [10:56:23] SMF on z-dat-s3-a is OK: OK - all services online [10:56:23] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:56:34] SMF on z-dat-s7-a is OK: OK - all services online [10:56:34] MySQL slave on z-dat-s7-a is OK: Uptime: 1560794 Threads: 7 Questions: 367134343 Slow queries: 58259 Opens: 2664536 Flush tables: 1 Open tables: 6814 Queries per second avg: 235.222 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 232 [10:56:34] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 226.000000 [10:56:44] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:56:54] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [10:56:54] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:57:04] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:04:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:07:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:07:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:11:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:29:38] hello all [11:35:55] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.127930/1.10, alarm hl:np_load_long=0.814453/1.55, alarm hl:mem_free=17822.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.127930/1.00, alarm hl:np_load_long=0.814453/1.50, alarm hl:mem_free=17822.000000M/300M, alarm hl:available=1/0 [11:37:04] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [11:42:04] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:35] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:05] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:34] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [11:43:34] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [11:43:43] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:53] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [11:44:06] SMF on z-dat-s6-a is OK: OK - all services online [11:44:06] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 262.000000 [11:44:06] SMF on z-dat-s7-a is OK: OK - all services online [11:44:06] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:44:13] MySQL on z-dat-s3-a is OK: Uptime: 1131240 Threads: 16 Questions: 1445438511 Slow queries: 98617 Opens: 10239000 Flush tables: 1 Open tables: 16384 Queries per second avg: 1277.746 [11:44:13] MySQL slave on z-dat-s3-a is OK: Uptime: 1131240 Threads: 16 Questions: 1445438512 Slow queries: 98617 Opens: 10239000 Flush tables: 1 Open tables: 16384 Queries per second avg: 1277.746 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 253 [11:44:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 458598 MB (8% inode=44%): [11:44:34] NTP on hyacinth is OK: NTP OK: Offset 0.000337 secs [12:04:14] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:06:53] 3(commented) [OSM-5] hstore operator ? no longer uses indexes <10https://jira.toolserver.org/browse/OSM-5> (Marlen Caemmerer) [12:06:57] 3(commented) [OSM-3] Ptolemy Postgres crashed twice <10https://jira.toolserver.org/browse/OSM-3> (Marlen Caemmerer) [12:07:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:07:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:09:03] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=3.539062/1.10, alarm hl:np_load_long=1.061524/1.55, alarm hl:mem_free=18527.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=3.539062/1.00, alarm hl:np_load_long=1.061524/1.50, alarm hl:mem_free=18527.000000M/300M, alarm hl:available=1/0 [12:11:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:15:05] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [12:37:04] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.057617/1.00, alarm hl:np_load_long=0.937500/1.50, alarm hl:mem_free=18775.000000M/300M, alarm hl:available=1/0 [12:38:04] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [12:44:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 458539 MB (8% inode=44%): [12:54:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31259 MB (3% inode=99%): [12:59:02] @replag [12:59:02] DaBPunkt: s3-rr-a: 18s [-0.00 s/s]; s3-user: 18s [-0.00 s/s]; s7-rr-a: 10s [+0.00 s/s]; s7-user: 10s [+0.00 s/s] [13:04:13] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:07:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:07:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:11:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:45:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 458462 MB (8% inode=44%): [14:04:24] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:07:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:07:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:11:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:45:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 458366 MB (8% inode=44%): [14:48:52] 3(commented) [OSM-5] hstore operator ? no longer uses indexes <10https://jira.toolserver.org/browse/OSM-5> (Kai Krueger) [15:04:25] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:07:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:07:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:11:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:24:03] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.179688/1.10, alarm hl:np_load_long=0.888672/1.55, alarm hl:mem_free=18801.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.179688/1.00, alarm hl:np_load_long=0.888672/1.50, alarm hl:mem_free=18801.000000M/300M, alarm hl:available=1/0 [15:25:05] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [15:45:04] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.153320/1.10, alarm hl:np_load_long=0.967773/1.55, alarm hl:mem_free=18921.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.153320/1.00, alarm hl:np_load_long=0.967773/1.50, alarm hl:mem_free=18921.000000M/300M, alarm hl:available=1/0 [15:45:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 458618 MB (8% inode=44%): [15:49:04] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [15:57:23] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.046875/1.8, alarm hl:np_load_avg=0.984863/2.3, alarm hl:mem_free=208.000000M/300M, alarm hl:available=1/0 [16:00:41] [[User:Whym]] ! 10https://wiki.toolserver.org/w/index.php?diff=6849&oldid=5906&rcid=9024 * Whym * (+229) (/* Projects */ +2) [16:04:23] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:05:23] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [16:07:02] Tim Landscheidt * Re: [Toolserver-l] [Wikitech-l] 403: User account expired toolserver.org/~soxred93 [16:07:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:07:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:09:30] Hi all, do we have statistics on how often each of the gadgets is activated on the different projects? [16:11:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:14:05] nope? [16:18:04] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.166016/1.10, alarm hl:np_load_long=0.935547/1.55, alarm hl:mem_free=18702.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.166016/1.00, alarm hl:np_load_long=0.935547/1.50, alarm hl:mem_free=18702.000000M/300M, alarm hl:available=1/0 [16:19:04] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [16:45:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 458022 MB (8% inode=44%): [17:01:02] Andrei Cipu * [Toolserver-l] Image import from Europeana [17:03:04] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.108399/1.10, alarm hl:np_load_long=0.852539/1.55, alarm hl:mem_free=18678.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.108399/1.00, alarm hl:np_load_long=0.852539/1.50, alarm hl:mem_free=18678.000000M/300M, alarm hl:available=1/0 [17:04:05] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:04:23] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:07:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:07:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:11:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:45:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 457867 MB (8% inode=44%): [18:02:14] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.688476/1.10, alarm hl:np_load_long=1.048828/1.55, alarm hl:mem_free=18140.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.688476/1.00, alarm hl:np_load_long=1.048828/1.50, alarm hl:mem_free=18140.000000M/300M, alarm hl:available=1/0 [18:04:13] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [18:04:28] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:07:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:07:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:11:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:34:54] 3(created) [ACCAPP-475] Run my global interwiki bot as well as run some future local bots for simple.wiki; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-475> (Rob Sutherland) [18:45:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 457747 MB (8% inode=44%): [18:56:56] 3(commented) [MNT-1198] nightshade down because of missing disc <10https://jira.toolserver.org/browse/MNT-1198> (Andreas F. Borchert) [19:05:23] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:07:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:07:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:11:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:12:13] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.671875/1.10, alarm hl:np_load_long=1.341797/1.55, alarm hl:mem_free=18622.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.671875/1.00, alarm hl:np_load_long=1.341797/1.50, alarm hl:mem_free=18622.000000M/300M, alarm hl:available=1/0 [19:14:13] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [19:45:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 457621 MB (8% inode=44%): [19:53:13] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.055664/1.00, alarm hl:np_load_long=0.889648/1.50, alarm hl:mem_free=17478.000000M/300M, alarm hl:available=1/0 [19:55:14] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [20:02:04] @replag [20:02:04] DaBPunkt: s2-user: 11s [+0.01 s/s]; s3-rr-a: 14s [+0.01 s/s]; s3-user: 14s [+0.01 s/s] [20:05:23] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:07:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:07:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:11:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:31:24] I play with the nagios-config – just if somebody wonders [20:34:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:34:06] Sensors on yarrow is CRITICAL: NRPE: Command check_sensors not defined [20:34:28] Hi, do we have statistics on gadget usage (opt-in/opt-out) [20:34:39] I cannot find this in the toolserver DBs [20:36:02] dschwen: should be in the user-preference-tables [20:43:35] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.178711/1.10, alarm hl:np_load_long=0.854492/1.55, alarm hl:mem_free=17676.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.178711/1.00, alarm hl:np_load_long=0.854492/1.50, alarm hl:mem_free=17676.000000M/300M, alarm hl:available=1/0 [20:45:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 457466 MB (8% inode=44%): [20:46:41] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:47:01] APT on yarrow is CRITICAL: APT CRITICAL: 5 packages available for upgrade (3 critical updates). [20:47:11] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Unknown database nlwiki_p [20:47:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:47:21] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Unknown database dewiki_p [20:55:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 457448 MB (8% inode=44%): [20:56:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:56:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:06:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:07:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:08:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:25:08] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 457253 MB (8% inode=44%): [21:26:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:26:19] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [21:26:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:50:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=2.547363/1.8, alarm hl:np_load_avg=1.332031/2.3, alarm hl:mem_free=447.000000M/300M, alarm hl:available=1/0: longrun@willow exceedes load threshold: alarm hl:np_load_short=2.547363/1.9, alarm hl:np_load_long=1.060547/2.25, alarm hl:mem_free=447.000000M/200M, alarm hl:available=1/0 [21:51:37] where is that table? [21:51:39] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [21:51:43] not on commonswiki_p [21:51:57] there I only see prefstats [21:52:06] and that only contains boring stuff like user skins [21:52:36] still here DaBPunkt? [21:57:35] mysql> select * from user_properties_anonym where up_property LIKE "gadget-%" limit 1; [21:57:36] +-----------------------+----------+-------------------------+ [21:57:38] | up_property | up_value | ts_user_touched_cropped | [21:57:39] +-----------------------+----------+-------------------------+ [21:57:41] | gadget-AddInformation | 0 | 20100826 | [21:59:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.874023/1.8, alarm hl:np_load_avg=1.053711/2.3, alarm hl:mem_free=267.000000M/300M, alarm hl:available=1/0 [22:10:08] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36014 MB (8% inode=99%): [22:11:37] is a ruby-speaker here? [22:14:08] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23262 MB (5% inode=99%): [22:20:28] Environment on yarrow is OK: ok: temperature ok fan ok voltage ok chassis ok [22:24:08] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:24:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:24:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:24:48] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:25:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 456852 MB (8% inode=44%): [22:26:08] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63769 MB (15% inode=99%): [22:26:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:26:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:26:55] DaBPunkt: ERROR 1146 (42S02): Table 'commonswiki_p.user_properties_anonym' doesn't exist [22:27:07] which server? [22:27:21] sql commonswiki_p, no idea where that gets me [22:27:32] but shouldn't it be the "right One"(TM) ? [22:27:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.820312/1.8, alarm hl:np_load_avg=1.675293/2.3, alarm hl:mem_free=701.000000M/300M, alarm hl:available=1/0 [22:27:57] s4 [22:28:10] mysql> select @@hostname\G [22:28:10] *************************** 1. row *************************** [22:28:10] @@hostname: z-dat-s4-a [22:28:36] dschwen: taht depens what "right" is for you [22:28:39] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [22:28:41] The view seems to be missing. [22:28:51] On S4's commonswiki_p. [22:29:11] yeah [22:30:15] I can see it on s2 though [22:30:36] I saw Krinkle mention this at some point. [22:30:38] but if I have to start guessing servers I might as well forget about it [22:30:52] I never know which commons copy is up to date and which one is outdated [22:30:52] mention what exactly? [22:31:14] Krinkle: Didn't you mention that user_preferences[_anonym] was missing? [22:31:19] dschwen: the view is just missing, nothing is outdated, cool down [22:31:25] I'm cool [22:31:30] I'm cooler. [22:32:01] so you are saying all commonswiki_p databases on all the sX servers are up-to-date? [22:32:06] Joan: Yes [22:32:16] Joan: But that was solved [22:32:22] Check JIRA ticket for more info [22:32:25] Krinkle: Not on s4, apparently. [22:32:34] including s4 [22:32:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=2.094238/1.8, alarm hl:np_load_avg=1.991699/2.3, alarm hl:mem_free=573.000000M/300M, alarm hl:available=1/0: longrun@willow exceedes load threshold: alarm hl:np_load_short=2.094238/1.9, alarm hl:np_load_long=1.550781/2.25, alarm hl:mem_free=573.000000M/200M, alarm hl:available=1/0 [22:32:39] sorry then, I thought I ran into some problems a while ago because I caught an old out of date commons copy on one of the servers [22:32:42] @replag [22:32:43] DaBPunkt: s3-rr-a: 30s [+0.00 s/s]; s3-user: 30s [+0.00 s/s]; s6-rr-a: 12s [+0.00 s/s]; s6-user: 12s [+0.00 s/s] [22:32:47] https://toolserver.org/~krinkle/tmp/user_properties_anonym.php [22:32:52] dschwen: yes [22:32:52] look, it's working there on s4 [22:32:56] the problem is fixed [22:33:02] thanks! [22:33:06] https://toolserver.org/~krinkle/tmp/user_properties_anonym.php?debug=true [22:33:08] Load avg. on willow is WARNING: WARNING - load average: 14.29, 15.40, 12.43 [22:33:11] GlobalConfig::setDbConnect> 0.0052500: Connection to commonswiki-p.rrdb.toolserver.org set. [22:33:33] Working now. :-) [22:33:38] ok :) [22:33:39] Krinkle: dschwen uses commonswiki-p.userdb for some reasons… [22:34:11] dschwen: if you just need READINg, use "sql -r commonswiki_p" plesae [22:34:13] DaBPunkt: Is there a script to synchronize the views across the databases? [22:34:31] Or I guess I'm asking: how does a view disappear? [22:34:56] Joan: in this case a database was missing and so the view was never created [22:35:04] Ah. [22:35:08] so, what does ts_user_touched_cropped mean? The last time the setting was changed? [22:35:08] Load avg. on willow is OK: OK - load average: 12.94, 14.69, 12.54 [22:35:35] dschwen: no, the last time the user touched his/her account (cropped) [22:36:21] touched as in logged in? performed an action? or changed his preferences? [22:36:28] all of that [22:36:35] any of that? [22:36:37] ok [22:36:42] thanks! [22:36:45] np [22:39:02] Sumana Harihareswara * [Toolserver-l] Fwd: Re: Image import from Europeana [22:43:08] Load avg. on willow is WARNING: WARNING - load average: 15.03, 14.95, 13.53 [22:51:02] Maarten Dammers * Re: [Toolserver-l] Image import from Europeana [23:15:08] Load avg. on willow is OK: OK - load average: 11.95, 13.72, 14.96 [23:24:08] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:24:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:24:59] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:25:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:26:08] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 456527 MB (8% inode=44%): [23:26:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:27:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:43:03] Danny B. * [Toolserver-l] Dumps maintenance part 1. finished