[00:05:25] dschwen: Did you see any issues after the switch of the DB? Or is everything working fine for you? [00:06:42] dschwen: I see you have already changed the attribution. Great. Thanks [00:09:12] looks good so far [00:09:22] I have not thoroughly tested this [00:09:43] there is a secret 'purge' option for the client-side tiles [00:10:06] if you press 'p' the current viewport is re-generated from the DB [00:10:37] I didn't dare testing this, as you said the hkey indices are still building [00:10:51] didn't know if this affects the results or just the qery performance [00:11:08] It shouldn't effect the results just the performance [00:11:15] But the index build is now complete [00:11:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (370928 kB) free! [00:11:26] So I think everything should work just as before [00:11:48] Or if not, then I'll need to fix it... [00:12:31] looks ok to me [00:14:25] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2124.000000 [00:15:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2083 [00:18:26] Free Memory on turnera is WARNING: WARNING - 5.3% (443652 kB) free! [00:20:05] MySQL slave on rosemary is OK: Uptime: 14547761 Threads: 15 Questions: 6389038614 Slow queries: 1954211 Opens: 258319 Flush tables: 6 Open tables: 3841 Queries per second avg: 439.176 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1788 [00:20:26] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1783.000000 [00:21:25] Free Memory on turnera is OK: OK - 7.2% (601460 kB) free. [00:24:57] @replag [00:24:58] Danny_B|backup: s1-rr-a: 27m 29s [-0.46 s/s]; s1-user: 27m 29s [-0.46 s/s]; s2-user: 22s [+0.00 s/s]; s3-rr-a: 3d 5h 9m 52s [+0.95 s/s]; s3-user: 3d 5h 9m 52s [+0.95 s/s]; s6-rr-a: 2d 18h 51m 10s [+0.61 s/s]; s6-user: 2d 18h 51m 10s [+0.61 s/s]; s7-rr-a: 2d 18h 13m 55s [+0.83 s/s] [00:24:59] Danny_B|backup: s7-user: 2d 18h 13m 55s [+0.83 s/s] [00:25:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56137 MB (9% inode=99%): [00:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:33:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 528485 MB (9% inode=44%): [00:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:36:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 239047 [00:37:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:38:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 241402 [00:44:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 278971 [00:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:54:26] Free Memory on turnera is WARNING: WARNING - 5.6% (467432 kB) free! [00:56:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (389224 kB) free! [01:05:26] Free Memory on turnera is WARNING: WARNING - 6.2% (516664 kB) free! [01:15:59] Merlissimo: you know all labs users will have easy access to your email address? [01:16:09] just saying because you hid it... [01:17:00] i hid it on mediawiki because of spam bots [01:17:29] k... [01:20:05] jeremyb: and this email address is special to labs/gerrit of course [01:21:12] really? [01:21:22] doesn't look like it! [01:25:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55537 MB (9% inode=99%): [01:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:27:26] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417916 kB) free! [01:33:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 528380 MB (9% inode=44%): [01:36:25] Free Memory on turnera is WARNING: WARNING - 6.8% (569576 kB) free! [01:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:37:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 242309 [01:38:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 244376 [01:43:26] Free Memory on turnera is OK: OK - 7.1% (591532 kB) free. [01:44:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 282492 [01:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:25:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55238 MB (9% inode=99%): [02:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:34:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 526516 MB (9% inode=44%): [02:36:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:37:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:38:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 245775 [02:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 247336 [02:44:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 285996 [02:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:25:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54652 MB (8% inode=99%): [03:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:30:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109156 MB (11% inode=99%): [03:34:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:34:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:35:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 525700 MB (9% inode=44%): [03:36:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:38:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 249251 [03:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 250650 [03:40:30] @replag [03:40:35] Hazard-SJ: s2-user: 1m 23s [+0.01 s/s]; s3-rr-a: 3d 8h 19m 44s [+0.97 s/s]; s3-user: 3d 8h 19m 44s [+0.97 s/s]; s5-rr-a: 19s [+0.00 s/s]; s5-user: 19s [+0.00 s/s]; s6-rr-a: 2d 21h 37m 48s [+0.85 s/s]; s6-user: 2d 21h 37m 51s [+0.85 s/s]; s7-rr-a: 2d 21h 16m 0s [+0.93 s/s] [03:40:36] Hazard-SJ: s7-user: 2d 21h 16m 0s [+0.93 s/s] [03:44:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:45:35] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 289469 [03:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:59:25] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:07:25] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (364924 kB) free! [04:09:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106890 MB (10% inode=99%): [04:10:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107028 MB (11% inode=99%): [04:11:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106756 MB (10% inode=99%): [04:12:25] Free Memory on turnera is WARNING: WARNING - 5.8% (486248 kB) free! [04:14:25] Free Memory on turnera is OK: OK - 7.3% (613640 kB) free. [04:26:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54610 MB (8% inode=99%): [04:26:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:26:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:31:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107716 MB (11% inode=99%): [04:34:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:35:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 526376 MB (9% inode=44%): [04:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:39:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 252692 [04:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 253794 [04:41:26] Free Memory on turnera is WARNING: WARNING - 6.0% (504412 kB) free! [04:41:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106880 MB (10% inode=99%): [04:42:25] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (408448 kB) free! [04:44:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 292939 [04:49:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:52:26] Free Memory on turnera is WARNING: WARNING - 5.3% (443088 kB) free! [05:02:24] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (323744 kB) free! [05:05:25] Ryan Lane * Re: [Toolserver-l] Reasons for not migrating to Tool Lab-OSM [05:07:25] Mike Dupont * Re: [Toolserver-l] Reasons for not migrating to Tool Lab-OSM [05:07:26] Free Memory on turnera is OK: OK - 7.5% (624828 kB) free. [05:17:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107878 MB (11% inode=99%): [05:26:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54526 MB (8% inode=99%): [05:27:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:27:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:33:26] Free Memory on turnera is CRITICAL: CRITICAL - 1.8% (149596 kB) free! [05:34:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 526191 MB (9% inode=44%): [05:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:38:25] Free Memory on turnera is WARNING: WARNING - 6.1% (513672 kB) free! [05:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:39:54] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 256006 [05:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 256827 [05:42:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (230992 kB) free! [05:44:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 296464 [05:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:05:15] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 10043 MB (0% inode=93%): [06:17:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106607 MB (10% inode=99%): [06:18:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107512 MB (11% inode=99%): [06:26:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54442 MB (8% inode=99%): [06:27:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:27:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:34:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:35:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 526061 MB (9% inode=44%): [06:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:39:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 259426 [06:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 259798 [06:42:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.7% (224328 kB) free! [06:44:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 300025 [06:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:50:45] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36221 MB (8% inode=99%): [07:06:04] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 78854 MB (8% inode=98%): [07:06:15] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 78854 MB (8% inode=98%): [07:06:25] Free Memory on turnera is WARNING: WARNING - 5.2% (437564 kB) free! [07:07:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (387728 kB) free! [07:23:25] aude * Re: [Toolserver-l] Reasons for not migrating to Tool Lab-OSM [07:26:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54323 MB (8% inode=99%): [07:27:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:27:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:32:26] Free Memory on turnera is WARNING: WARNING - 5.3% (443092 kB) free! [07:34:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:35:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (403828 kB) free! [07:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 525933 MB (9% inode=44%): [07:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:36:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:39:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 262813 [07:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 263244 [07:44:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:45:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 303567 [07:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:26] Free Memory on turnera is WARNING: WARNING - 6.1% (507012 kB) free! [08:10:26] Free Memory on turnera is OK: OK - 7.9% (663668 kB) free. [08:27:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54270 MB (8% inode=99%): [08:27:24] Free Memory on turnera is WARNING: WARNING - 5.7% (476696 kB) free! [08:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:34:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:35:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 525566 MB (9% inode=44%): [08:36:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:36:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 266839 [08:40:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 266281 [08:45:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:45:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 307113 [08:49:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:56:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1932 [08:56:25] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1938.000000 [09:03:25] Free Memory on turnera is WARNING: WARNING - 6.4% (532780 kB) free! [09:28:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54089 MB (8% inode=99%): [09:28:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:28:25] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:31:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (408664 kB) free! [09:32:25] Free Memory on turnera is WARNING: WARNING - 6.2% (516676 kB) free! [09:34:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:35:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:35:26] Bryan Tong Minh * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [09:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 525402 MB (9% inode=44%): [09:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:37:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:37:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 270433 [09:40:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 269640 [09:45:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 310658 [09:50:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:56:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2755 [09:56:26] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2745.000000 [09:57:25] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (416616 kB) free! [10:07:26] Free Memory on turnera is WARNING: WARNING - 6.1% (513248 kB) free! [10:22:05] MySQL slave on rosemary is OK: Uptime: 14583880 Threads: 13 Questions: 6413730555 Slow queries: 1963452 Opens: 260463 Flush tables: 6 Open tables: 3863 Queries per second avg: 439.782 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1767 [10:22:26] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1747.000000 [10:28:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53902 MB (8% inode=99%): [10:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:34:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:35:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 525218 MB (9% inode=44%): [10:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:37:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 273997 [10:40:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 273010 [10:45:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:45:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 314201 [10:50:25] Free Memory on turnera is OK: OK - 7.1% (591436 kB) free. [10:50:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:00:25] Free Memory on turnera is WARNING: WARNING - 7.0% (589428 kB) free! [11:02:26] Free Memory on turnera is OK: OK - 7.2% (606916 kB) free. [11:28:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53726 MB (8% inode=99%): [11:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:35:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:36:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.2% (355504 kB) free! [11:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 525014 MB (9% inode=44%): [11:36:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:36:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:37:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:37:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 277417 [11:40:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 276494 [11:45:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:45:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 317738 [11:47:28] @replag [11:50:17] jem-: s1-rr-a: 1m 6s [-0.14 s/s]; s1-user: 1m 6s [-0.14 s/s]; s2-user: 1m 34s [+0.00 s/s]; s3-rr-a: 3d 16h 17m 30s [+0.98 s/s]; s3-user: 3d 16h 20m 17s [+0.98 s/s]; s6-rr-a: 3d 5h 13m 7s [+0.95 s/s]; s6-user: 3d 5h 13m 7s [+0.95 s/s]; s7-rr-a: 3d 4h 57m 46s [+0.97 s/s] [11:50:18] jem-: s7-user: 3d 4h 57m 46s [+0.97 s/s] [11:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:07:26] Load avg. on adenia is WARNING: WARNING - load average: 17.04, 15.04, 9.11 [12:19:25] Load avg. on adenia is OK: OK - load average: 8.08, 13.87, 12.25 [12:28:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53542 MB (8% inode=99%): [12:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:35:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:36:25] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (305636 kB) free! [12:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 524868 MB (9% inode=44%): [12:36:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:37:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:38:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 280635 [12:41:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 280064 [12:45:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:45:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 321295 [12:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:53:26] Free Memory on turnera is OK: OK - 7.1% (593476 kB) free. [13:03:26] Free Memory on turnera is WARNING: WARNING - 5.1% (427268 kB) free! [13:05:25] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (406916 kB) free! [13:25:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105457 MB (10% inode=99%): [13:27:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108172 MB (11% inode=99%): [13:28:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53321 MB (8% inode=99%): [13:28:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:28:25] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:33:26] Free Memory on turnera is WARNING: WARNING - 6.3% (524024 kB) free! [13:34:26] Free Memory on turnera is OK: OK - 7.1% (592080 kB) free. [13:35:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 524627 MB (9% inode=44%): [13:36:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:37:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 283894 [13:41:26] Free Memory on turnera is WARNING: WARNING - 6.3% (527900 kB) free! [13:41:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 283482 [13:45:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 324831 [13:46:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:16] hello all [13:51:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:54:26] John * Re: [Toolserver-l] [Wikitech-l] Toolserver is not opening for GU:WP [14:08:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (383964 kB) free! [14:12:26] Free Memory on turnera is WARNING: WARNING - 5.1% (429512 kB) free! [14:21:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (386576 kB) free! [14:28:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53288 MB (8% inode=99%): [14:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:34:39] @replag [14:34:39] Alchimista: s1-rr-a: 4m 32s [+0.02 s/s]; s1-user: 4m 32s [+0.02 s/s]; s2-user: 19s [-0.01 s/s]; s3-rr-a: 3d 19h 2m 1s [+0.98 s/s]; s3-user: 3d 19h 2m 1s [+0.98 s/s]; s6-rr-a: 3d 7h 41m 6s [+0.90 s/s]; s6-user: 3d 7h 41m 6s [+0.90 s/s]; s7-rr-a: 3d 7h 34m 50s [+0.96 s/s] [14:34:40] Alchimista: s7-user: 3d 7h 34m 50s [+0.96 s/s] [14:35:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:36:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 524325 MB (9% inode=44%): [14:36:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:38:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:38:26] Alchimista * Re: [Toolserver-l] [Wikitech-l] Toolserver is not opening for GU:WP [14:39:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:40:25] DaB. * [Toolserver-l] Switching of SGE-arch-default at 9. October [14:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 287151 [14:41:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 286866 [14:43:25] Hersfold Wikipedia * Re: [Toolserver-l] Switching of SGE-arch-default at 9. October [14:45:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 328362 [14:46:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:51:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:14:26] Merlissimo * Re: [Toolserver-l] Switching of SGE-arch-default at 9. October [15:21:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.6% (220160 kB) free! [15:28:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52804 MB (8% inode=99%): [15:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:35:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 523935 MB (9% inode=44%): [15:36:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:37:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:38:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 290583 [15:41:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 290247 [15:45:25] Free Memory on turnera is WARNING: WARNING - 5.4% (449236 kB) free! [15:45:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 331915 [15:46:26] Free Memory on turnera is OK: OK - 7.1% (598036 kB) free. [15:47:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:53:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.4% (285536 kB) free! [16:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:28:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:29:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52537 MB (8% inode=99%): [16:36:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:36:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 523632 MB (9% inode=44%): [16:36:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:38:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:39:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 293920 [16:42:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 293623 [16:45:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 335448 [16:47:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:53:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.4% (205076 kB) free! [17:28:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:28:25] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:29:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52254 MB (8% inode=99%): [17:36:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:37:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 521596 MB (9% inode=44%): [17:37:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:38:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:39:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 297124 [17:42:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 296986 [17:45:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 338997 [17:47:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:53:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (302332 kB) free! [18:05:15] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 9813 MB (0% inode=93%): [18:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:29:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52065 MB (8% inode=99%): [18:36:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:37:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 521038 MB (9% inode=44%): [18:37:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:38:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 300407 [18:42:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 300403 [18:45:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 342551 [18:47:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:51:46] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35540 MB (8% inode=99%): [18:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:53:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.3% (279996 kB) free! [19:06:05] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 78481 MB (8% inode=98%): [19:06:14] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 78478 MB (8% inode=98%): [19:13:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106524 MB (10% inode=99%): [19:16:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108451 MB (11% inode=99%): [19:18:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106783 MB (10% inode=99%): [19:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:29:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51869 MB (8% inode=99%): [19:30:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107456 MB (11% inode=99%): [19:36:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:37:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 520902 MB (9% inode=44%): [19:37:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:37:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:38:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:39:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:39:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 303735 [19:41:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2154 [19:41:13] @replag [19:41:26] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2171.000000 [19:41:34] jem-: s1-rr-a: 36m 10s [+0.10 s/s]; s1-user: 36m 10s [+0.10 s/s]; s3-rr-a: 4d 3m 31s [+0.98 s/s]; s3-user: 4d 3m 33s [+0.98 s/s]; s6-rr-a: 3d 12h 23m 11s [+0.92 s/s]; s6-user: 3d 12h 23m 28s [+0.92 s/s]; s7-rr-a: 3d 12h 21m 35s [+0.93 s/s]; s7-user: 3d 12h 21m 35s [+0.93 s/s] [19:42:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106924 MB (10% inode=99%): [19:42:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 303762 [19:43:25] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107251 MB (11% inode=99%): [19:44:26] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [19:44:35] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2342.000000 [19:47:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:47:55] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2110.000000 [19:50:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 346351 [19:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:53:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.4% (281712 kB) free! [19:53:55] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1734.000000 [20:18:25] Free Memory on turnera is WARNING: WARNING - 5.1% (428692 kB) free! [20:19:26] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (419436 kB) free! [20:19:35] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1785.000000 [20:20:05] MySQL slave on rosemary is OK: Uptime: 14619761 Threads: 19 Questions: 6436213977 Slow queries: 1974630 Opens: 264064 Flush tables: 6 Open tables: 3873 Queries per second avg: 440.240 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1767 [20:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:29:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51685 MB (8% inode=99%): [20:36:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 520488 MB (9% inode=44%): [20:37:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:38:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:39:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 307080 [20:42:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 307138 [20:48:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:50:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 349894 [20:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:19:25] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (298584 kB) free! [21:21:17] I will now restart z-dat-s6-a [21:28:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:28:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:29:55] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [21:30:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51644 MB (8% inode=99%): [21:37:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:37:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 520245 MB (9% inode=44%): [21:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:37:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:38:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:38:55] MySQL on z-dat-s6-a is OK: Uptime: 142 Threads: 14 Questions: 2888 Slow queries: 3 Opens: 117 Flush tables: 1 Open tables: 110 Queries per second avg: 20.338 [21:38:59] ok, its back [21:39:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 309912 [21:40:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:41:54] @replag [21:41:55] DaBPunkt: s2-user: 1m 44s [+0.00 s/s]; s2-user-c: 59s [+0.00 s/s]; s3-rr-a: 4d 2h 2m 3s [+0.98 s/s]; s3-user: 4d 2h 2m 3s [+0.98 s/s]; s5-user-c: 58s [+0.00 s/s]; s6-rr-a: 3d 13h 54m 10s [+0.75 s/s]; s6-user: 3d 13h 54m 10s [+0.75 s/s]; s7-rr-a: 3d 14h 12m 39s [+0.92 s/s] [21:41:56] DaBPunkt: s7-user: 3d 14h 12m 39s [+0.92 s/s] [21:42:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 310399 [21:44:48] @replag [21:44:49] DaBPunkt: s1-rr-a: 14s [-0.29 s/s]; s1-user: 14s [-0.29 s/s]; s2-user: 20s [-0.48 s/s]; s3-rr-a: 4d 2h 4m 38s [+0.89 s/s]; s3-user: 4d 2h 4m 39s [+0.90 s/s]; s6-rr-a: 3d 13h 31m 56s [-7.69 s/s]; s6-user: 3d 13h 31m 56s [-7.69 s/s]; s7-rr-a: 3d 14h 14m 23s [+0.60 s/s] [21:44:50] DaBPunkt: s7-user: 3d 14h 14m 23s [+0.60 s/s] [21:45:01] better [21:48:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:51:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 353475 [21:51:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:18:26] Free Memory on turnera is WARNING: WARNING - 5.2% (436260 kB) free! [22:20:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (399032 kB) free! [22:28:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:28:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:30:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51452 MB (8% inode=99%): [22:37:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 520012 MB (9% inode=44%): [22:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:37:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:39:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 307163 [22:40:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:43:25] Free Memory on turnera is WARNING: WARNING - 5.1% (423280 kB) free! [22:43:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 313282 [22:45:25] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420272 kB) free! [22:47:26] Free Memory on turnera is WARNING: WARNING - 5.1% (426960 kB) free! [22:48:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:52:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 356931 [22:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:02:39] DaBPunkt: remember the python error related to unicode? Wasn't solved? [23:04:56] which error [23:04:57] ? [23:05:34] the one with python 2.6, related to unicode and ascii. i thought you've aplyed a patch or something [23:05:43] yes [23:06:13] it was supose to work both on linux and solaris? [23:06:28] it worked on solaris before [23:07:05] yes, but seems that is back again, i'm not sure why [23:08:14] I just ran version.py (from pywikibot) and it said ok [23:08:25] Alchimista: there is no python2.6 on the solaris boxes AFAIR [23:09:44] just checked nightshade, worked fine too [23:10:04] DaBPunkt: strange, it's already 2.7.1. must be a new bug [23:10:55] but lets see if anyone else also complains, i've noticed it in some old scripts wich use the print statment, i'm changing them to not use the print, and seems to be fine [23:13:56] legoktm: the unicode bug that happens when you "print " something [23:14:52] Hm. I'm not too familiar with that one, is that filed somewhere? [23:17:12] legoktm: the one with this thead signal: UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 13: ordinal not in range(128) [23:17:13] lool [23:18:16] Oh [23:18:27] I know that one all too well :( [23:19:42] nacht ts [23:23:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (367068 kB) free! [23:23:34] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1954.000000 [23:24:05] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1946 [23:28:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:29:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:31:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51234 MB (8% inode=99%): [23:37:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 518633 MB (9% inode=44%): [23:37:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:37:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 307746 [23:40:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:40:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:43:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 306728 [23:48:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:52:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:52:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 357524 [23:55:26] Free Memory on turnera is WARNING: WARNING - 5.2% (432624 kB) free! [23:59:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (380508 kB) free!