[00:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 146317 [00:01:11] Ryan_Vesey: Depends which database you're qurying. [00:01:12] querying [00:01:17] I assume en.wiki? [00:01:20] yes [00:01:34] Probably care about that first value. [00:01:36] s1-rr-a [00:01:41] thank you [00:01:43] So you're likely 15 hours behind. [00:01:58] It's not catching up as quickly as it should be. [00:02:08] Might be overloaded... [00:02:29] That would affect bots on the wiki, correct? [00:02:37] Some of them, sure. [00:02:48] Some bots don't query the databases. [00:02:56] Or they query older data where it doesn't matter. [00:03:05] it --> replag [00:03:12] You understand what replag is? [00:03:17] I guess that's probably a better question. [00:03:40] I realize that it slows things down, and occasionally shuts down the watchlist [00:03:43] but that's about it [00:04:00] Replag doesn't slow things down, really. [00:04:07] Replag is short for "replication lag." [00:04:14] To be more specific, WP 1.0 bot updates tables for article classes, and has been updating for the past 4 days; however, the new information hasn't been included [00:04:23] The databases live in one place. The Toolserver has its own copies. Sometimes these copies get lagged. [00:04:33] The copy can't go fast enough given the rate of edits, basically. [00:04:47] How stale is the data? [00:04:49] your comment that it queries older data may be exactly our problem; however, why would it not grab the data that is only 15 hours past [00:04:57] If it's four days old, that doesn't sound like replag. [00:04:58] there hasn't been a correct update in 4 days [00:05:16] hmm, then we're back to square 1 [00:05:17] Well, one of the databases is corrupt. [00:05:24] Can you link the page in question? [00:05:36] http://en.wikipedia.org/wiki/Wikipedia_talk:Version_1.0_Editorial_Team/Index#Bot_not_updating_table_properly.3F [00:05:52] and the table is http://en.wikipedia.org/wiki/User:WP_1.0_bot/Tables/Custom/Roads-1 [00:06:04] Are you a WP:ROADS person? [00:06:14] Nope [00:06:33] https://en.wikipedia.org/w/index.php?title=User:WP_1.0_bot/Tables/Custom/Roads-1&action=history [00:06:38] Just curious. Most of the roads folks are nuts. [00:06:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54168 [00:06:44] It looks like the page is updating every day. [00:06:46] Including today. [00:06:53] You think the data is wrogn? [00:06:54] wrong [00:07:11] I have no clue, but the Roads people think so [00:07:34] If it's not replag, I have to assume that the class parameter hasn't been updated [00:07:47] Well, as I said, one of the databases is corrupt. [00:08:07] So the Toolserver copies the enwiki database, right? But it actually copies it twice. [00:08:10] True story. [00:08:13] One of the servers' copies is corrupt. [00:08:19] Which is probably this problem. [00:08:34] Think you can explain that on the talk page? [00:08:57] Oh, heh. [00:09:00] I didn't even read the talk page. [00:09:01] It's clearly an error actually [00:09:16] http://en.wikipedia.org/wiki/Category:GA-Class_U.S._road_transport_articles 739 GAs [00:09:20] and the table only shows 723 [00:10:19] Oh, an example. [00:10:24] Let's test the theory, then. [00:10:40] The category membership counts are always a bit fuzzy. [00:11:16] I get 730 from rosemary. [00:11:30] And 728 from thyme. [00:11:34] thyme is the corrupt one. [00:12:42] I still have the page open from when I did the manual update, is there a way to see which database it pulled from? [00:13:10] Only if the tool you're using tells you. [00:13:21] enwiki_p is on thyme and rosemary. [00:13:25] thyme's copy is corrupt. [00:13:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40066 MB (9% inode=99%): [00:14:08] Category membership counts on category description pages have always been a bit fuzzy, particularly if there are over 200 category members. [00:14:20] The 739 figure has probably been wrong for ages. [00:14:37] gotcha [00:14:41] But if the table isn't updating with accurate counts and it used to, that might be the DB corruption. [00:14:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 146690.000000 [00:15:04] https://jira.toolserver.org/browse/TS-1414 is the relevant ticket. [00:16:51] Well, I'll point the ticket out, beyond that I have to run, hopefully someone knows how to fix it, thanks for your assistance [00:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:34:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:38:30] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58864 MB (14% inode=99%): [00:39:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54398.000000 [00:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:44:34] hello! is there a way to redirect PHP errors to a given file so that I can get just my own errors? [00:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575266 MB (10% inode=48%): [01:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 147329 [01:01:26] inductiveload: What's wrong with filtering the log? [01:04:52] should i just do something like "grep inductiveload /var/log/userlog"? [01:07:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54678 [01:08:35] i wasn't sure if i should be keeping my junk to myself ;-) [01:14:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147454.000000 [01:24:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:31:30] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39127 MB (9% inode=99%): [01:31:33] inductiveload: Yes, grep is fine. [01:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:34:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54754.000000 [01:42:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23432 MB (5% inode=99%): [01:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:56:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575226 MB (10% inode=48%): [01:58:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45047 MB (11% inode=99%): [02:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 147185 [02:05:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 30428 MB (7% inode=99%): [02:07:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54686 [02:12:40] /sql on thyme is WARNING: DISK WARNING - free space: /sql 175132 MB (18% inode=99%): [02:14:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147463.000000 [02:17:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 54380 MB (13% inode=99%): [02:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54475.000000 [02:45:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:57:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573273 MB (10% inode=48%): [03:00:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 148097 [03:07:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54311 [03:14:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 148307.000000 [03:24:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54087.000000 [03:45:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:50:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106086 MB (10% inode=98%): [03:50:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106059 MB (10% inode=98%): [03:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:54:30] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 108457 MB (11% inode=98%): [03:54:39] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 108437 MB (11% inode=98%): [03:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573305 MB (10% inode=48%): [04:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 149133 [04:08:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 53928 [04:15:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 149366.000000 [04:24:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53821.000000 [04:45:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573179 MB (10% inode=48%): [05:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 149821 [05:08:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54570 [05:16:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 150111.000000 [05:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:24:50] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 57942 MB (4% inode=98%): [05:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55081.000000 [05:45:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:48:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 105164 MB (10% inode=98%): [05:48:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 105161 MB (10% inode=98%): [05:53:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 108774 MB (11% inode=98%): [05:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:54:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 108069 MB (11% inode=98%): [05:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573011 MB (10% inode=48%): [06:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 150563 [06:08:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55336 [06:16:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 150844.000000 [06:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:41:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55735.000000 [06:45:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 572919 MB (10% inode=48%): [07:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 151013 [07:08:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56340 [07:16:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 151059.000000 [07:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:41:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56823.000000 [07:46:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:53:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:58:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 572770 MB (10% inode=48%): [08:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 151146 [08:08:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57422 [08:16:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 151386.000000 [08:17:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106503 MB (10% inode=98%): [08:17:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106510 MB (10% inode=98%): [08:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:38:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107714 MB (11% inode=98%): [08:38:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107712 MB (11% inode=98%): [08:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:41:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58034.000000 [08:46:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:48:49] Load avg. on willow is WARNING: WARNING - load average: 14.25, 15.95, 13.80 [08:49:50] Load avg. on willow is OK: OK - load average: 11.42, 14.82, 13.54 [08:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:58:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 572677 MB (10% inode=48%): [09:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 151733 [09:02:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41434 MB (10% inode=99%): [09:08:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58683 [09:16:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 152028.000000 [09:25:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:26:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58247 MB (14% inode=99%): [09:26:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106352 MB (10% inode=98%): [09:27:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106268 MB (10% inode=98%): [09:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:33:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107190 MB (11% inode=98%): [09:33:39] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107190 MB (11% inode=98%): [09:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:41:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59277.000000 [09:41:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106371 MB (10% inode=98%): [09:41:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106369 MB (10% inode=98%): [09:46:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:58:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 572596 MB (10% inode=48%): [10:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 153001 [10:09:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59788 [10:16:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 153339.000000 [10:25:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:29:30] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107527 MB (11% inode=98%): [10:29:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107445 MB (11% inode=98%): [10:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:34:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:36:30] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106773 MB (10% inode=98%): [10:36:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106790 MB (10% inode=98%): [10:40:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:41:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60351.000000 [10:47:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:53:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:58:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 572484 MB (10% inode=48%): [11:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 153765 [11:09:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 60849 [11:16:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 153961.000000 [11:25:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:41:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61150.000000 [11:47:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:53:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:58:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 572388 MB (10% inode=48%): [12:00:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 154429 [12:02:59] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:07:49] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:09:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61613 [12:16:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 154730.000000 [12:21:39] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107784 MB (11% inode=98%): [12:22:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 108635 MB (11% inode=98%): [12:25:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:34:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:41:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61489.000000 [12:47:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:53:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127223 MB (20% inode=99%): [12:53:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 572241 MB (10% inode=48%): [13:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 155532 [13:09:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61338 [13:16:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 155727.000000 [13:24:30] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22036 MB (5% inode=99%): [13:25:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:35:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:39:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 46211 MB (11% inode=99%): [13:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:41:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61177.000000 [13:47:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:53:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127014 MB (20% inode=99%): [13:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 572142 MB (10% inode=48%): [14:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 156060 [14:09:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61147 [14:13:40] /sql on thyme is WARNING: DISK WARNING - free space: /sql 174612 MB (18% inode=99%): [14:16:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156409.000000 [14:26:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:35:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:37:30] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 104741 MB (10% inode=98%): [14:37:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 104698 MB (10% inode=98%): [14:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:41:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60965.000000 [14:47:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:53:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126816 MB (20% inode=99%): [14:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571977 MB (10% inode=48%): [15:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 156894 [15:09:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 110071 MB (11% inode=98%): [15:09:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61158 [15:09:39] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 108337 MB (11% inode=98%): [15:16:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157225.000000 [15:25:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:35:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:42:09] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [15:47:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:49:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106401 MB (10% inode=98%): [15:50:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106390 MB (10% inode=98%): [15:53:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126598 MB (20% inode=99%): [15:53:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:55:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107823 MB (11% inode=98%): [15:55:39] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 108143 MB (11% inode=98%): [15:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571421 MB (10% inode=48%): [16:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 157662 [16:09:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61370 [16:16:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157955.000000 [16:26:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:42:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61640.000000 [16:47:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126360 MB (20% inode=99%): [16:54:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571329 MB (10% inode=48%): [17:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 158425 [17:09:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61757 [17:16:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 158672.000000 [17:22:04] Dr. Trigon * Re: [Toolserver-l] Quota [17:24:50] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 58826 MB (4% inode=98%): [17:26:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:40:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:42:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61866.000000 [17:47:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:53:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126150 MB (20% inode=99%): [17:54:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:59:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571251 MB (10% inode=48%): [18:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 159617 [18:10:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61959 [18:16:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 159943.000000 [18:26:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:41:38] hello all [18:42:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61850.000000 [18:47:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:48:04] Dr. Trigon * Re: [Toolserver-l] Install OpenCV libraries and python bindings [18:48:22] hi DaBPunkt [18:48:48] hallo Toto_Azero ­how is the MMP going? [18:49:13] DaBPunkt: it's working well for the moment ;) [18:49:30] good [18:50:04] if the ts-community is not coming up with rules during July, I will write something together arround august [18:50:17] ok [18:50:56] because TS-users don't seems to be in a hurry :p [18:53:04] Dr. Trigon * Re: [Toolserver-l] Quota [18:54:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125926 MB (20% inode=99%): [18:54:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571129 MB (10% inode=48%): [19:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 160530 [19:10:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61856 [19:16:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 160755.000000 [19:28:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:39:06] is there a way to find out how often a tool of mine is accesed? [19:39:15] so I decide if it's worthy to improve it or not? [19:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:42:29] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [19:44:10] magister: it is worth to improve if 1 user is using it, isn't it? ;) [19:44:32] DaBPunkt: I doulbt even 1 is using it ;) [19:44:36] magister: but to answer your question: grep the access-log [19:44:51] I'll try that, thank you [19:44:57] like ggrep -c "myurl" /var(log/http/access [19:45:12] aight [19:45:27] what server is that? [19:45:30] not on willow [19:47:23] no, of course not [19:47:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:47:37] wolfsbane and ortelius are the web-servers [19:54:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125684 MB (20% inode=99%): [19:54:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571022 MB (10% inode=48%): [20:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 161219 [20:10:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62341 [20:16:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 161412.000000 [20:28:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:42:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62415.000000 [20:44:59] Load avg. on hyacinth is WARNING: WARNING - load average: 16.36, 15.33, 11.90 [20:45:40] Load avg. on z-dat-s3-a is WARNING: WARNING - load average: 17.46, 15.75, 12.21 [20:45:50] Load avg. on z-dat-s6-a is WARNING: WARNING - load average: 17.33, 15.75, 12.22 [20:45:50] Load avg. on z-dat-s4-a is WARNING: WARNING - load average: 17.33, 15.75, 12.22 [20:45:50] Load avg. on z-dat-s7-a is WARNING: WARNING - load average: 17.30, 15.75, 12.22 [20:47:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:50:58] Load avg. on hyacinth is OK: OK - load average: 14.14, 14.94, 12.91 [20:51:39] Load avg. on z-dat-s3-a is OK: OK - load average: 13.55, 14.68, 12.91 [20:51:50] Load avg. on z-dat-s6-a is OK: OK - load average: 13.47, 14.65, 12.91 [20:51:50] Load avg. on z-dat-s4-a is OK: OK - load average: 13.47, 14.65, 12.91 [20:51:50] Load avg. on z-dat-s7-a is OK: OK - load average: 13.45, 14.64, 12.91 [20:54:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125429 MB (20% inode=99%): [20:54:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570868 MB (10% inode=48%): [21:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 161857 [21:10:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62650 [21:17:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162084.000000 [21:29:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:34:13] ggrep? [21:34:43] Brooke: the gnu grep [21:34:45] Oh, not a typo. That's a real flavor of grep. [21:34:49] Ah. [21:35:12] And grep is Solaris' version? [21:35:15] the solaris grep (just "grep") is *differend* [21:35:37] for example is supports no recursive search [21:36:30] Right, right. Solaris makes life painful. [21:36:41] What happened to nightshade? [21:36:48] Is it coming back with Debian? [21:36:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:37:27] Brooke: yes. The installation is nearly done. I just need to patch python again and than the linux-servers can be used [21:37:33] somewhen this week [21:37:37] Cool. :-) [21:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:42:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62790.000000 [21:47:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:54:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125178 MB (20% inode=99%): [21:54:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570863 MB (10% inode=48%): [22:00:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 162283 [22:05:03] ok Soapy, try runes again [22:10:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62749 [22:17:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162586.000000 [22:28:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:40:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:43:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62469.000000 [22:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:49:56] vim is funny on debian, less polished [22:54:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124937 MB (20% inode=99%): [22:54:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570776 MB (10% inode=48%): [23:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 162717 [23:01:04] Platonides * Re: [Toolserver-l] Quota [23:10:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62064 [23:13:24] nacht ts [23:17:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162837.000000 [23:30:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:33:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:36:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:41:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:43:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62004.000000 [23:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:54:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124740 MB (20% inode=99%): [23:55:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 569422 MB (10% inode=48%):