[00:02:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89001 [00:10:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:10:57] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100539.000000 [00:12:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:13:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:13:16] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 210561 MB (3% inode=67%): [00:16:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:16:43] nacht ts [00:16:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:17:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97816 MB (16% inode=99%): [00:17:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:22:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46494.000000 [00:27:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53206 MB (5% inode=99%): [00:36:56] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10808 [00:51:22] how can I understand which is the itwiki sql server? [00:54:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87040.000000 [00:55:13] fale: one sec [00:55:35] Betacommand: thanks :) [00:55:55] https://wiki.toolserver.org/view/Wiki_server_assignments [00:56:23] Betacommand: thanks :) [00:56:29] fale: np [00:57:33] Betacommand: may I ask you why it seems that the database itwiki_p is not present on s2, as it should be based on the link you passed me? [00:59:35] fale: what are you trying to do? [01:00:01] Betacommand: an sql query on the itwiki replicated db to extract a list of pages [01:00:09] sql itwiki_p [01:01:42] sql itwiki_p "query"> results.txt [01:02:01] avoids the whole server assignment issues [01:02:05] Betacommand: thanks a lot :) I was trying to use the mysql command and I was pretty lost :D. Thanks for the heads up :) [01:02:23] fale: np [01:03:56] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87160 [01:06:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:10:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:11:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102525.000000 [01:12:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:13:16] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 212300 MB (3% inode=67%): [01:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:16:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:17:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:18:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97584 MB (16% inode=99%): [01:21:05] DaB. * Re: [Toolserver-l] database lag issues [01:22:05] Load avg. on ortelius is WARNING: WARNING - load average: 17.92, 15.68, 10.21 [01:22:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48651.000000 [01:36:05] Load avg. on ortelius is OK: OK - load average: 3.83, 12.95, 13.28 [01:37:55] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11962 [01:50:44] is there a way to use queries longher than 600 seconds? [01:55:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88063.000000 [02:03:56] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:04:40] SLOW_OK [02:06:41] Dispenser: that allows me to move trom 60 to 600 seconds, but no more than that [02:06:57] @replag [02:06:57] Dispenser: s1-rr-a-wd: 1m 32s [-]; s1-user: 1d 38m 50s [-]; s1-user-wd: 55s [-]; s2-rr: 3h 41m 48s [-]; s2-user: 3h 41m 48s [-]; s2-user-c: error; s2-user-wd: 14h 6m 20s [-]; s3-user: 11m 8s [-] [02:06:59] Dispenser: s3-user-wd: 39s [-]; s4-user-wd: 1d 5h 4m 17s [-]; s5-user: 3m 37s [-]; s5-user-c: error; s6-user-wd: 20s [-]; s7-user-wd: 21s [-] [02:07:15] use a RR server [02:07:39] Dispenser: like itwiki-p.rrdb.toolserver.org? [02:08:20] fale: what kind of query are you running it might be possible to optimize it [02:09:29] Betacommand: a query that searches all pages containing copyrighted screenshots that do not contain specific templates [02:10:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:10:35] fale: care to pastebin it? [02:10:42] Betacommand: and the query is here: http://it.wikipedia.org/wiki/Wikipedia:Elenchi_generati_offline/Screenshot#Screenshot_in_assenza_di_template [02:11:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104744.000000 [02:11:53] fale: I would break that into multiple queries and just do the post processing yourself [02:12:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:12:44] Betacommand: one to extract the pages with ScreenShots, one to extract the pages with those template and then merge the two lists? [02:13:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:13:16] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 211920 MB (3% inode=67%): [02:13:46] let me look in to the query [02:13:48] MySQL optimizes `tl_title IN (...list...)` poorly [02:14:09] Betacommand: http://pastebin.com/LAezf9dN [02:14:42] Dispenser: what would be optimized better? [02:16:01] I'm rewriting it [02:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:16:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:16:49] Dispenser: thanks :) [02:17:13] fale: you could make the whole last part faster by using page_id instead of AND (page_namespace, page_title) NOT IN [02:17:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:18:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95508 MB (15% inode=99%): [02:18:24] Betacommand: thanks :) [02:20:35] I would have just done several crude requests, probably would have been a little bit of a disc hog, but that would only be for a short time [02:21:20] then just done the unions, intersections and other comparisons after the fact [02:22:11] I compare a list of all files on enwiki in less than 3 seconds using pythons list features, doing so in mysql would take a lot longer [02:22:16] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51065.000000 [02:23:03] Betacommand: I see :) [02:25:13] Load avg. on ortelius is WARNING: WARNING - load average: 19.93, 18.58, 14.42 [02:30:22] fale: http://pastebin.com/GqawPrfB 1734 rows in set (53.54 sec) [02:30:59] Dispenser: thanks a lot :) [02:32:14] Load avg. on ortelius is OK: OK - load average: 10.21, 14.84, 14.27 [02:33:00] Dispenser: I did not thought it was so optimizable... I'm going to work to optimize all the other queries a lot more taking example by this one you did [02:33:43] EXPLAIN help me alot, too bad it doesn't work anymore :-( [02:33:54] BTW, it would be nice to have some Toolserver wiki page or whatever where queries to be optimized and query optimizers could meet :-). [02:34:15] Dispenser: Do you know why? [02:34:47] Something changed in the upgrade, need to bug nosy about it again [02:35:16] It would be really useful if it could be enabled. [02:35:55] what was it? [02:36:18] http://dev.mysql.com/doc/refman/5.5/en/using-explain.html [02:36:35] thanks :) [02:36:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:36:58] basically the join method the order of those joins and the statistics used [02:38:55] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10206 [02:41:03] Once the database stats were messed up, EXPLAIN told me it run the query in reverse (Horribly slow), and fixed it with STRAIGHT_JOIN [02:56:21] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91669.000000 [03:02:45] scfc_de: There's... https://wiki.toolserver.org/view/Queries [03:04:05] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:10:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:11:15] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106995.000000 [03:12:30] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:13:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 209934 MB (3% inode=67%): [03:13:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:16:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:17:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:17:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:18:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97181 MB (15% inode=99%): [03:22:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51884.000000 [03:33:37] Susan: Some of these queries look familiar :-). [03:36:05] :-) [03:37:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:39:23] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4000 [03:41:23] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3575 [03:45:23] MySQL slave on z-dat-s2-b is OK: Uptime: 701354 Threads: 11 Questions: 1052089576 Slow queries: 11420 Opens: 11926217 Flush tables: 1 Open tables: 256 Queries per second avg: 1500.83 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [03:56:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95285.000000 [04:04:24] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:10:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:11:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106530.000000 [04:12:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:13:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 209781 MB (3% inode=67%): [04:13:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:16:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:17:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:17:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:18:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97067 MB (15% inode=99%): [04:20:15] I think numerical string handling has changed in MySQL [04:22:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53123.000000 [04:33:22] nvm, logic mistake [04:33:25] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1805 [04:34:24] MySQL slave on z-dat-s2-b is OK: Uptime: 704296 Threads: 10 Questions: 1057204624 Slow queries: 11453 Opens: 11955935 Flush tables: 1 Open tables: 256 Queries per second avg: 1501.79 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [04:37:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:56:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98901.000000 [04:58:53] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (308796 kB) free! [05:04:24] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:08:12] Free Memory on turnera is OK: OK - 90.7% (7599068 kB) free. [05:10:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:12:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 209299 MB (3% inode=67%): [05:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96912 MB (15% inode=99%): [05:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54585.000000 [05:42:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:56:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102507.000000 [06:05:10] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:11:40] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103739.000000 [06:12:09] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 208986 MB (3% inode=67%): [06:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96690 MB (15% inode=99%): [06:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56328.000000 [06:42:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:56:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106107.000000 [07:05:10] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:11:40] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104972.000000 [07:12:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 208652 MB (3% inode=67%): [07:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96554 MB (15% inode=99%): [07:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58057.000000 [07:42:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:50:00] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2087 [07:56:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109708.000000 [08:05:10] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:11:40] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106674.000000 [08:12:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 208651 MB (3% inode=67%): [08:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96406 MB (15% inode=99%): [08:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60842.000000 [08:42:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:50:00] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2416 [08:56:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108127.000000 [09:05:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106850 [09:06:00] MySQL slave on z-dat-s2-b is OK: Uptime: 720593 Threads: 12 Questions: 1082356946 Slow queries: 11767 Opens: 12087492 Flush tables: 1 Open tables: 256 Queries per second avg: 1502.36 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1790 [09:11:40] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108298.000000 [09:12:09] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 207956 MB (3% inode=67%): [09:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:18:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96242 MB (15% inode=99%): [09:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62531.000000 [09:42:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:49:00] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2015 [09:56:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100721.000000 [10:03:00] MySQL slave on z-dat-s2-b is OK: Uptime: 724013 Threads: 12 Questions: 1088335739 Slow queries: 11888 Opens: 12106829 Flush tables: 1 Open tables: 256 Queries per second avg: 1503.199 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1798 [10:05:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98632 [10:11:40] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109726.000000 [10:12:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 207657 MB (3% inode=67%): [10:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96053 MB (15% inode=99%): [10:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63422.000000 [10:42:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:56:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91466.000000 [11:05:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90304 [11:11:39] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104869.000000 [11:12:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 207086 MB (3% inode=67%): [11:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:19:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95839 MB (15% inode=99%): [11:24:39] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65138.000000 [11:27:20] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2023.000000 [11:29:50] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1972.000000 [11:42:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:52:20] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1900.000000 [11:56:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80597.000000 [12:05:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 79038 [12:11:40] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95045.000000 [12:12:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 206734 MB (3% inode=67%): [12:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95626 MB (15% inode=99%): [12:19:50] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1925 [12:20:20] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1934.000000 [12:21:20] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [12:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66080.000000 [12:27:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52599 MB (5% inode=99%): [12:29:50] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3401.000000 [12:35:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [12:42:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:46:20] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [12:47:01] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3235.000000 [12:52:20] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3290.000000 [12:52:30] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [12:53:52] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3660 [12:56:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74774.000000 [13:05:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73913 [13:11:40] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90934.000000 [13:12:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:12:40] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4697.000000 [13:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 206402 MB (3% inode=67%): [13:17:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:18:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [13:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95463 MB (15% inode=99%): [13:21:21] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5054.000000 [13:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68223.000000 [13:30:20] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3586.000000 [13:34:00] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1956 [13:35:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4306.000000 [13:41:00] MySQL slave on z-dat-s2-b is OK: Uptime: 737094 Threads: 9 Questions: 1106149277 Slow queries: 12168 Opens: 12230948 Flush tables: 1 Open tables: 256 Queries per second avg: 1500.689 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [13:42:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:53:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000 [13:53:52] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4270 [13:56:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69415.000000 [14:03:41] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3584.000000 [14:03:51] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3546 [14:05:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 69000 [14:11:40] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90524.000000 [14:12:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:13:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:14:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:14:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 205905 MB (3% inode=67%): [14:16:40] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1794.000000 [14:16:50] MySQL slave on z-dat-s5-b is OK: Uptime: 428354 Threads: 2 Questions: 114819777 Slow queries: 8668 Opens: 522153 Flush tables: 1 Open tables: 255 Queries per second avg: 268.48 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1735 [14:17:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:17:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:18:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:19:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95263 MB (15% inode=99%): [14:20:31] Free Memory on damiana is WARNING: WARNING - 5.8% (486168 kB) free! [14:21:20] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5446.000000 [14:23:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68354.000000 [14:23:30] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (404468 kB) free! [14:36:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5317.000000 [14:36:48] hello all [14:37:50] Free Memory on damiana is OK: OK - 92.0% (7710228 kB) free. [14:38:35] Hi DaBPunkt [14:43:28] @replag [14:43:29] DaBPunkt: s1-rr-a-wd: 11m 36s [-]; s1-user: 18h 19m 49s [-]; s1-user-wd: 1h 36m 17s [-]; s2-rr: 8m 46s [-]; s2-user: 8m 46s [-]; s2-user-c: error; s2-user-wd: 19h 8m 31s [-]; s3-user: 24s [-] [14:43:30] DaBPunkt: s3-user-wd: 2h 9m 54s [-]; s4-user-wd: 1d 1h 25m 8s [-]; s5-user: 5m 2s [-]; s5-user-c: 2w 2d 17h 56m 35s [-]; s6-user-wd: 1h 40m 42s [-]; s7-user-wd: 1h 14m 5s [-] [14:53:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4608.000000 [14:56:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65436.000000 [15:05:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64635 [15:11:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91070.000000 [15:12:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:12:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 205630 MB (3% inode=67%): [15:14:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:18:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:18:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:19:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:19:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94990 MB (15% inode=99%): [15:21:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6424.000000 [15:23:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70137.000000 [15:35:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94923 MB (15% inode=99%): [15:35:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:35:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:35:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:35:50] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6410.000000 [15:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6460.000000 [15:42:27] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94905 MB (15% inode=99%): [15:43:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94903 MB (15% inode=99%): [15:43:40] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:43:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6325.000000 [15:48:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94898 MB (15% inode=99%): [15:48:48] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:48:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:48:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:48:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6216.000000 [15:52:26] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94901 MB (15% inode=99%): [15:52:31] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:52:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:52:31] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:52:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6166.000000 [15:55:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94846 MB (15% inode=99%): [15:55:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:55:36] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:55:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6125.000000 [15:55:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4189.000000 [15:57:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61204.000000 [16:00:21] Just in case anybody wonders: I work on the SVN [16:01:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94820 MB (15% inode=99%): [16:01:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:01:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:01:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:01:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6056.000000 [16:01:27] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4156.000000 [16:06:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 60144 [16:07:06] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [16:23:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [16:23:20] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94754 MB (15% inode=99%): [16:23:26] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59743 [16:23:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:23:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:23:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5913.000000 [16:23:37] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3783.000000 [16:39:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94702 MB (15% inode=99%): [16:39:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:39:08] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59800 [16:39:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:39:08] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:39:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5770.000000 [16:39:17] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [16:41:17] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3592.000000 [16:42:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94660 MB (15% inode=99%): [16:42:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:42:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59801 [16:42:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:42:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:42:57] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5692.000000 [16:42:57] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3572.000000 [16:44:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94655 MB (15% inode=99%): [16:44:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:44:20] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 59772 [16:44:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:44:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:44:30] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5706.000000 [16:44:31] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3581.000000 [16:46:31] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3604.000000 [16:50:10] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [16:52:10] PING on amaranth is WARNING: PING WARNING - Packet loss = 20%, RTA = 182.00 ms [16:54:00] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 136.00 ms [17:15:59] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 172.00 ms [17:17:10] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 165.00 ms [17:18:20] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 148.00 ms [17:40:20] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 179.00 ms [17:43:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 204474 MB (3% inode=67%): [17:43:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71377.000000 [17:43:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:43:49] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93735.000000 [17:43:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:44:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8470.000000 [17:44:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94403 MB (15% inode=99%): [17:44:20] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55712 [17:44:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:44:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:44:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:44:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:44:30] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6988.000000 [17:44:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55657.000000 [17:44:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:47:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4903.000000 [17:50:09] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [18:16:20] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 154.00 ms [18:23:20] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 191.00 ms [18:37:20] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 149.00 ms [18:43:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:43:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 96634.000000 [18:43:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:44:00] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10306.000000 [18:44:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94126 MB (15% inode=99%): [18:44:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49544 [18:44:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:44:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:44:25] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:44:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74497.000000 [18:44:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203998 MB (3% inode=67%): [18:44:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49549.000000 [18:44:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:45:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8583.000000 [18:48:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6527.000000 [18:50:08] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [18:51:09] PING on amaranth is WARNING: PING WARNING - Packet loss = 20%, RTA = 161.00 ms [18:54:00] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 150.00 ms [19:08:59] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 183.00 ms [19:14:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:14:58] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 142.00 ms [19:24:50] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2058.000000 [19:25:20] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2077.000000 [19:42:20] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 180.00 ms [19:43:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:43:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99939.000000 [19:43:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12625.000000 [19:43:59] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 160.00 ms [19:44:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93890 MB (15% inode=99%): [19:44:20] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39455 [19:44:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:44:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:44:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:44:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77391.000000 [19:44:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203696 MB (3% inode=67%): [19:44:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39384.000000 [19:44:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:45:30] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10745.000000 [19:45:59] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 157.00 ms [19:47:20] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 159.00 ms [19:48:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8678.000000 [19:50:09] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10218 [19:53:10] PING on amaranth is WARNING: PING WARNING - Packet loss = 20%, RTA = 187.00 ms [19:57:50] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [19:58:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [20:10:20] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 186.00 ms [20:12:19] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 157.00 ms [20:17:59] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 112.00 ms [20:25:19] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3490.000000 [20:34:50] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3599.000000 [20:43:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:43:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13895.000000 [20:44:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93610 MB (15% inode=99%): [20:44:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:44:20] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32304 [20:44:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:44:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:44:30] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:44:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79041.000000 [20:44:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203328 MB (3% inode=67%): [20:44:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102784.000000 [20:44:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32274.000000 [20:44:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:44:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:45:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11762.000000 [20:46:09] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3582 [20:48:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8951.000000 [20:50:20] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1778.000000 [20:52:05] Sumana Harihareswara * [Toolserver-l] Submit a talk to Open Source Bridge by Mar 23rd, get WMF subsidy [20:52:10] MySQL slave on z-dat-s3-a is OK: Uptime: 759163 Threads: 24 Questions: 1034676072 Slow queries: 44996 Opens: 9168043 Flush tables: 1 Open tables: 16384 Queries per second avg: 1362.916 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1642 [20:53:59] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 174.00 ms [20:55:00] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 141.00 ms [21:05:00] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 173.00 ms [21:07:20] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 171.00 ms [21:08:19] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 153.00 ms [21:10:05] Marc A. Pelletier * [Toolserver-l] This week's update, now with 7.3% more levity! [21:15:19] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 176.00 ms [21:35:50] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3315.000000 [21:43:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:44:00] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14586.000000 [21:44:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93326 MB (15% inode=99%): [21:44:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:44:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80439.000000 [21:44:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203124 MB (3% inode=67%): [21:44:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105240.000000 [21:44:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30516.000000 [21:44:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:44:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:45:26] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30501 [21:45:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:45:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11821.000000 [21:48:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9067.000000 [22:14:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:22:50] wikidata replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1795.000000 [22:43:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:44:28] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12251.000000 [22:44:28] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93081 MB (15% inode=99%): [22:44:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:44:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82454.000000 [22:44:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 202594 MB (3% inode=67%): [22:44:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30904.000000 [22:44:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107590.000000 [22:44:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:45:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30899 [22:45:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:45:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7554.000000 [22:48:46] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5622.000000 [23:20:47] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3546.000000 [23:21:40] @replag [23:21:40] DaBPunkt: s1-rr-a-wd: 28s [-0.44 s/s]; s1-user: 8h 23m 6s [-0.05 s/s]; s1-user-wd: 2h 45m 1s [-0.61 s/s]; s2-rr: 3m 43s [+0.02 s/s]; s2-user: 3m 43s [+0.02 s/s]; s2-user-c: error; s2-user-wd: 23h 5m 2s [+0.45 s/s]; s3-user-wd: 3h 8m 18s [-0.63 s/s] [23:21:41] DaBPunkt: s4-user-wd: 1d 6h 14m 39s [+0.64 s/s]; s5-user-c: 2w 3d 2h 34m 47s [+1.00 s/s]; s6-user-wd: 1h 9m 32s [-1.11 s/s]; s7-user-wd: 58m 5s [-0.74 s/s] [23:21:48] Free Memory on turnera is WARNING: WARNING - 6.4% (535928 kB) free! [23:22:06] "s5-user-c: 2w 3d"????? [23:28:47] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3595.000000 [23:31:47] Free Memory on turnera is OK: OK - 7.3% (610220 kB) free. [23:43:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:44:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:44:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8815.000000 [23:44:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92850 MB (15% inode=99%): [23:44:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:44:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80859.000000 [23:44:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 197809 MB (3% inode=67%): [23:45:03] @replag [23:45:03] DaBPunkt: s1-rr-a-wd: 34s [+0.00 s/s]; s1-user: 7h 44m 13s [-1.66 s/s]; s1-user-wd: 2h 26m 16s [-0.80 s/s]; s2-user-c: error; s2-user-wd: 22h 27m 2s [-1.63 s/s]; s3-user-wd: 2h 43m 50s [-1.05 s/s]; s4-user-wd: 1d 6h 29m 1s [+0.61 s/s]; s5-user-c: 2w 3d 2h 58m 9s [+1.00 s/s] [23:45:04] DaBPunkt: s6-user-wd: 35m 5s [-1.47 s/s]; s7-user-wd: 33m 33s [-1.05 s/s] [23:45:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27858.000000 [23:45:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:45:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109743.000000 [23:45:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:45:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27828 [23:45:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:47:47] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1790.000000 [23:47:47] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1739.000000 [23:47:47] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (373432 kB) free!