[00:03:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:03:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:05:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (411568 kB) free! [00:06:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:07:58] Free Memory on turnera is WARNING: WARNING - 5.4% (455792 kB) free! [00:08:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:08:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:16:58] Free Memory on turnera is OK: OK - 7.2% (599648 kB) free. [00:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:25:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 7335 MB (10% inode=95%): [00:38:47] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39087 MB (9% inode=99%): [00:38:59] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73574 MB (7% inode=98%): [00:39:58] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6242 MB (20% inode=93%): [00:39:58] / on wolfsbane is WARNING: DISK WARNING - free space: / 6242 MB (20% inode=93%): [00:40:08] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73562 MB (7% inode=98%): [00:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:48:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:54:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:56:19] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [00:57:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89573 MB (14% inode=99%): [01:03:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:03:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:06:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:08:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:08:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:10:46] nacht ts [01:17:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:17:25] DaB. * Re: [Toolserver-l] /mnt/user-store is away [01:18:59] Free Memory on turnera is CRITICAL: CRITICAL - 2.5% (211464 kB) free! [01:26:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 6879 MB (9% inode=95%): [01:30:58] Free Memory on turnera is WARNING: WARNING - 5.3% (447208 kB) free! [01:33:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.2% (271472 kB) free! [01:39:59] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6076 MB (20% inode=93%): [01:39:59] / on wolfsbane is WARNING: DISK WARNING - free space: / 6076 MB (20% inode=93%): [01:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:48:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:54:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:56:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [01:57:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89506 MB (14% inode=99%): [02:03:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:04:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:06:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:08:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:08:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:26:18] / on damiana is CRITICAL: DISK CRITICAL - free space: / 6097 MB (8% inode=95%): [02:39:58] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5886 MB (19% inode=93%): [02:39:58] / on wolfsbane is WARNING: DISK WARNING - free space: / 5886 MB (19% inode=93%): [02:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:48:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:54:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:56:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [02:57:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89466 MB (14% inode=99%): [03:04:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:04:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:06:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:09:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:09:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:17:58] /tmp on wolfsbane is OK: DISK OK - free space: / 9462 MB (31% inode=93%): [03:17:58] / on wolfsbane is OK: DISK OK - free space: / 9462 MB (31% inode=93%): [03:18:58] Free Memory on turnera is WARNING: WARNING - 6.8% (573152 kB) free! [03:22:58] Free Memory on turnera is OK: OK - 7.1% (594544 kB) free. [03:26:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 3185 MB (4% inode=95%): [03:36:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (326132 kB) free! [03:39:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:48:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:54:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:56:19] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [03:57:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89400 MB (14% inode=99%): [03:59:48] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:04:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:04:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:06:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:10:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:10:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:26:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 5477 MB (7% inode=95%): [04:47:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:48:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:54:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:56:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [04:57:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89330 MB (14% inode=99%): [05:05:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:05:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:06:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:07:08] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (232364 kB) free! [05:10:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:10:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:26:18] / on damiana is CRITICAL: DISK CRITICAL - free space: / 4635 MB (6% inode=95%): [05:31:08] Free Memory on turnera is WARNING: WARNING - 5.1% (423556 kB) free! [05:32:09] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (390200 kB) free! [05:47:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:48:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:52:09] Free Memory on turnera is OK: OK - 8.1% (677040 kB) free. [05:54:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:56:19] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [05:57:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89263 MB (14% inode=99%): [05:59:09] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (250732 kB) free! [06:05:09] Free Memory on turnera is OK: OK - 8.7% (724868 kB) free. [06:05:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:05:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:06:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:10:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:10:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:17:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:26:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 3024 MB (4% inode=95%): [06:47:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:49:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:54:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:56:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [06:57:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89181 MB (14% inode=99%): [07:05:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:05:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:06:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:10:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:10:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:11:39] I seem to have let my ts account expire :( [07:11:43] any way I can get it back? [07:11:46] without re-applying? [07:13:23] file a bug on jira [07:13:33] :( ok [07:14:14] I think all the admins are sleeping [07:14:26] oh and i can run stuff for you yuvipanda [07:15:20] legoktm: woo, sweet [07:15:38] its not going to be long though right? I want to sleep in 45 minutes ish [07:15:46] legoktm: mobile team wants http://toolserver.org/~emijrp/wlm/stats.php for mobile, so am modifying it [07:15:51] no, should take me ~30 mins at most [07:15:54] thanks! [07:15:57] gimme a couple of mins [07:16:03] Ah ok [07:17:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:23:36] legoktm: how good is your SQL? [07:23:49] i'm trying to check for multiple categories and my SQL looks a little... inelegant [07:24:43] er I'm not the best…I can try though [07:26:11] actally let's try run this [07:26:13] legoktm: https://gist.github.com/3831969 [07:26:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 1604 MB (2% inode=95%): [07:26:22] put that on your public folder and run? [07:26:55] You need pywikipedia? ok [07:27:35] er. [07:27:54] I dont have his javascript modules, do you want to reference absolute paths so you can pick his up? [07:28:05] let me update [07:28:26] actually that should be fine for now, I think. [07:28:30] i just want to check if the data works fine [07:28:32] also line 37 [07:28:33] ok [07:28:55] what's your ts username? i'll update path [07:28:58] legoktm [07:29:06] let me make sure it can access pywikipedia [07:29:36] updated gist. do create wlm directory :) [07:31:49] I just found a second way that does not involve a self join [07:32:29] legoktm: any luck? [07:32:45] let me see [07:33:00] im setting up pywikipedia [07:33:41] sweet [07:34:07] https://toolserver.org/~legoktm/cgi-bin/wlm/mobilestats.py [07:35:10] legoktm: can you update from latest code in the gist? [07:35:16] sure [07:35:16] alsoc reate public_html/wlm directory? [07:35:47] done [07:35:47] ok [07:36:02] done [07:36:09] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (395932 kB) free! [07:37:06] legoktm: can you tell me if there's a stats.php in /home/legoktm/public_html/wlm [07:37:08] Free Memory on turnera is WARNING: WARNING - 5.5% (458548 kB) free! [07:37:09] ? [07:37:24] nope [07:37:55] public_html/wlm is set to 777 btw [07:38:14] legoktm: update from gist again? [07:38:18] sure [07:38:48] done [07:39:18] should i be running the mobilestats.py from the command line? [07:39:24] or is it supposed to be cgi? [07:39:34] shouldn't matter, I guess? [07:39:38] try running it from the commandline? [07:39:44] i think that should give you an error message at least [07:40:24] ah [07:40:25] oursql.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'JOIN categorylinks c2 ON c2.cl_from=page_id WHERE c2.cl_to=?' at line 7", None) [07:40:52] thought so [07:40:55] legoktm: update again? [07:41:12] doing [07:41:13] holygoddamn is github down? [07:41:51] if it is, http://pastebin.com/XwP0UngB is latest code [07:42:03] i grabbed something... [07:42:07] let me just grab it again [07:42:43] it printed "unitedstates" [07:42:50] no errors? [07:42:51] amongst a pywikipedia warning [07:42:54] nope [07:42:54] is the php file there now? [07:42:55] but [07:42:58] it hasnt ended [07:43:01] ah [07:43:05] that's alright :) [07:43:07] it's working yay [07:43:09] ok [07:43:22] it doesn't do what I want yet though. [07:43:26] nothing in public_html/wlm [07:43:28] * yuvipanda goes to add one more clause [07:43:36] yeah it should print 'done unitedstates' soon [07:43:38] (hopefully) [07:43:43] ok [07:46:03] legoktm: stop it and try https://gist.github.com/3831969 again [07:46:18] should be faster [07:46:38] just "unitedstates" [07:46:52] ah [07:46:57] done unitedstates [07:46:58] 1076 files [07:47:01] sweeet [07:47:09] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (379112 kB) free! [07:47:10] and something in wlm! [07:47:35] why can't I see it? :| [07:47:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:47:41] https://toolserver.org/~legoktm/public_html/wlm/stats.php [07:47:42] wrong url? [07:47:55] ah yup [07:48:01] just /wlm [07:48:16] I see some stuff [07:48:17] :D [07:48:26] works [07:48:26] yay [07:48:30] let me run this for everything now [07:48:45] ok what do i have to do for that? [07:48:52] nothing let me update the code [07:48:55] ok [07:49:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:50:29] legoktm: update from https://gist.github.com/3831969 again? [07:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:51:44] ok running [07:53:25] \o/ [07:53:56] done [07:54:12] whee [07:54:14] https://toolserver.org/~legoktm/wlm/stats.php [07:54:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:54:46] looks pretty cool [07:55:19] indeed :D [07:55:24] i should fix a few more links, I think [07:57:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89053 MB (14% inode=99%): [07:57:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [07:59:52] yuvipanda: I would like to go to sleep in a few minutes if possible [08:01:44] 5 minutes? one final thing [08:02:05] sure [08:03:47] legoktm: update from https://gist.github.com/3831969 again? [08:04:57] ok [08:05:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:05:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:05:36] hit an error [08:05:54] ou? [08:06:08] http://pastebin.com/K2suEACG [08:06:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:06:40] aah, got it [08:09:09] Free Memory on turnera is WARNING: WARNING - 6.0% (501716 kB) free! [08:09:44] legoktm: update now? [08:10:04] sure [08:10:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:10:44] same error, this time line 262 [08:10:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:11:25] did that update correctly? I just counted and it looks ok... [08:12:34] ill try again [08:13:09] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (362176 kB) free! [08:13:10] yup. line 262, im using this copy: https://raw.github.com/gist/3831969/5bf4d108e40d5b139a44a203c4cc94070b22f8ec/mobilestats.py [08:14:24] legoktm: try now? [08:14:44] ok [08:15:14] done [08:15:19] and no errors :) [08:15:23] sweet! [08:16:17] legoktm: thanks a lot! :) [08:16:22] no problem [08:16:27] im going to sleep now, ill probably be up in about 6-7 hours, shoot me an email if something blows up :P [08:16:33] awesome [08:16:34] thanks! [08:17:08] Free Memory on turnera is WARNING: WARNING - 5.1% (428836 kB) free! [08:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:22:28] does anybody know which style file is used for the mapnik database at the toolserver? [08:26:18] / on damiana is CRITICAL: DISK CRITICAL - free space: / 179 MB (0% inode=95%): [08:26:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [08:29:25] Peter Wendorff * [Toolserver-l] osm2pgsql style used by the toolserver [08:44:09] Free Memory on turnera is WARNING: WARNING - 5.8% (489884 kB) free! [08:47:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:49:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:54:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:57:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88905 MB (14% inode=99%): [08:57:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [08:59:09] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (363196 kB) free! [09:02:42] where was /mnt/user-store gone? [09:03:38] Tanvir: ? ^ [09:04:11] or Brooke ? [09:05:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:05:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:07:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:10:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:10:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:14:08] Free Memory on turnera is WARNING: WARNING - 5.5% (459692 kB) free! [09:14:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109071 MB (11% inode=99%): [09:15:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 104662 MB (10% inode=99%): [09:17:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:23:09] Free Memory on turnera is OK: OK - 7.1% (598604 kB) free. [09:26:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [09:26:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [09:37:09] Free Memory on turnera is WARNING: WARNING - 5.6% (466640 kB) free! [09:38:37] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:09] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 72947 MB (7% inode=98%): [09:42:08] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (403120 kB) free! [09:44:09] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:44:19] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:44:48] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:48:09] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:49:08] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38547 MB (9% inode=99%): [09:49:18] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [09:49:18] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 72939 MB (7% inode=98%): [09:49:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:51:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:51:18] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1990 [09:51:18] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2072 [09:54:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:55:17] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [09:57:17] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88735 MB (14% inode=99%): [09:58:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [10:04:08] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73002 MB (7% inode=98%): [10:05:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:05:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:07:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:08:17] MySQL on z-dat-s3-a is OK: Uptime: 1140 Threads: 5 Questions: 2934 Slow queries: 2 Opens: 146 Flush tables: 1 Open tables: 136 Queries per second avg: 2.573 [10:08:17] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3322 [10:11:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:15:17] MySQL slave on z-dat-s6-a is OK: Uptime: 349504 Threads: 9 Questions: 65073549 Slow queries: 25525 Opens: 673078 Flush tables: 1 Open tables: 2806 Queries per second avg: 186.188 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1657 [10:17:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:18:17] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3618 [10:24:08] Free Memory on turnera is WARNING: WARNING - 5.6% (472864 kB) free! [10:26:27] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [10:27:17] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3415 [10:27:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [10:29:08] Free Memory on turnera is OK: OK - 7.2% (602868 kB) free. [10:35:17] MySQL slave on z-dat-s7-a is OK: Uptime: 78911 Threads: 10 Questions: 28144107 Slow queries: 1482 Opens: 232818 Flush tables: 1 Open tables: 1270 Queries per second avg: 356.656 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1745 [10:35:17] MySQL slave on z-dat-s3-a is OK: Uptime: 2760 Threads: 22 Questions: 947781 Slow queries: 28 Opens: 20320 Flush tables: 1 Open tables: 10021 Queries per second avg: 343.398 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1781 [10:39:26] Merlissimo * Re: [Toolserver-l] How to have qsub mail output? [10:48:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:49:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:51:27] Load avg. on adenia is WARNING: WARNING - load average: 15.67, 14.34, 8.61 [10:52:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:52:28] Load avg. on adenia is OK: OK - load average: 12.79, 13.69, 8.74 [10:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:57:17] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88587 MB (14% inode=99%): [10:58:17] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [11:04:08] Free Memory on turnera is WARNING: WARNING - 5.4% (452604 kB) free! [11:05:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:05:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:07:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:11:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:28] Load avg. on adenia is WARNING: WARNING - load average: 15.78, 15.36, 13.38 [11:16:08] Free Memory on turnera is OK: OK - 7.5% (624432 kB) free. [11:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:26:28] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [11:28:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [11:48:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:49:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:52:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:58:17] Free Memory on turnera is WARNING: WARNING - 5.4% (453240 kB) free! [11:58:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88429 MB (14% inode=99%): [11:58:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [11:59:09] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (390900 kB) free! [12:01:54] perhaps anyone knows now: I would like to know how exactly the osm_mapnik database is imported to set up a corresponding database locally for testing [12:04:02] hello all [12:04:17] Free Memory on turnera is OK: OK - 7.5% (625992 kB) free. [12:05:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:05:27] DaBPunkt: hi [12:05:32] can you help me please? [12:05:34] Hi DaBPunkt. Small question in case you know by accident: how is the osm_mapnik database imported exactly to the toolserver? would like to have the same database schema here for testing. [12:05:56] jongleur: no idea sorry. Speak with apmon [12:06:03] k, thanks [12:06:04] matanya: that depens :) [12:06:07] where is /mnt/user-store? all dumps disappeared [12:06:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:07:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:07:41] matanya: http://lists.wikimedia.org/pipermail/toolserver-announce/2012-October/000528.html [12:08:28] oh! [12:08:39] ah, and a second (quite different) question - but probably anyone else can help here, too: is it possible to connect to the toolserver jira from eclipse as a task repository? any trick I have to apply to get a working connection? [12:08:52] I didn't do the connection between this post to this issue [12:09:24] jongleur: I tried that once and it worked. But our jira has a auth-problem at the moment so it may fail [12:09:34] so, DaBPunkt can I have a dump some where on the server? [12:09:39] k, then that might be the problem. [12:10:04] matanya: can't it wait a few hours until we build a work-around? [12:10:12] DaBPunkt: "auth-problem" = logs out every few minutes? :-) [12:10:35] scfc_de: as in "has no connection to crowd" [12:10:42] I guess it can, it is waiting since oct 1st [12:11:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:11:33] BTW, is it possible to copy a bot to wmflabs? [12:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:12:09] DaBPunkt: I've never seen that message, JIRA just logs me out every few minutes (< 10). [12:14:15] thanks DaBPunkt have a nice day [12:14:21] bye [12:15:07] / on wolfsbane is WARNING: DISK WARNING - free space: / 6211 MB (20% inode=93%): [12:15:07] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6211 MB (20% inode=93%): [12:16:08] / on wolfsbane is OK: DISK OK - free space: / 12450 MB (41% inode=93%): [12:16:08] /tmp on wolfsbane is OK: DISK OK - free space: / 12450 MB (41% inode=93%): [12:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:17:52] matanya: Yes, if it doesn't need replicated databases. Take a look at https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots and contact Petrb. [12:27:28] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [12:28:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [12:48:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:49:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:52:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:58:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88101 MB (14% inode=99%): [12:58:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [13:05:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:06:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:07:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:11:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:17:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:19:29] re [13:27:27] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [13:28:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [13:41:27] / on damiana is OK: DISK OK - free space: / 22658 MB (31% inode=95%): [13:49:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:49:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:52:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:54:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:58:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87929 MB (14% inode=99%): [13:58:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [14:05:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:06:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:07:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:10:36] DaBPunkt: so usr-store is back? [14:11:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:11:32] Danny_B|backup: when I conviced willow to mount it [14:11:44] yarrow and nightshade has it already [14:11:52] yeah, i see that nightshade sees it but willow not [14:11:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:12:43] hmm, nightshade does not have ttyallow :-/ [14:12:52] so i can't run a screen for mmp [14:13:15] looks like willow needs a reboot [14:13:53] uff, then let me know ahead [14:14:12] Danny_B|backup: I have to announce that anyway [14:15:23] is the problem finally fixed then? [14:16:06] giftpflanze: no. its a workarround [14:16:47] DaBPunkt: does ttyallow exist on linux? [14:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:24:09] Danny_B|backup: AFAIS screen does work fine without ttyallow [14:24:15] at linux [14:27:19] will try [14:27:27] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:27:57] great :( [14:28:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [14:32:27] Sun Grid Engine execd on ortelius is CRITICAL: Connection refused by host [14:36:18] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.029 second response time [14:41:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:49:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:49:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:52:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:52:15] usrstore back on willow [14:53:26] .oO(why have i ported my script to linux *cry*) [14:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:58:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87753 MB (14% inode=99%): [14:58:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [14:58:21] Danny_B|backup: sure? [14:58:38] just running some stuff there [14:59:13] or wait [14:59:18] wrong window [14:59:22] sorry for confusion [14:59:47] however it's deadly slow on nightshade [15:01:18] what? [15:01:47] user-store [15:02:06] in mc, it takes minutes to show the directory content [15:02:27] spinner is moving for minutes [15:02:36] and no update yet [15:03:45] for 8 minutes already [15:03:52] and still no response [15:04:24] i would say nightshade is slow in general [15:05:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:05:55] it never was the way it would not response for 10 mins [15:07:12] it's actually not nightshade but userstore accessed via nightshade [15:07:22] which does not response promptly [15:07:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:11:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:12:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:17:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:26:27] Danny_B|backup: is it better now? [15:27:13] (what you all do in the user-store in the first place anyway?) [15:28:24] ^^ [15:28:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [15:29:14] i use it for data which needs no backup [15:39:26] DaB. * [Toolserver-announce] user-store (temporary) back, willow will rebooted friday [15:41:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:49:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:50:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:53:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:54:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:58:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87584 MB (14% inode=99%): [15:58:19] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [16:05:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:08:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:12:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:28:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [16:36:44] Hallo! [16:36:52] Bin neu hier! [16:37:40] Wofür ist dieser Toolserver? [16:41:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:44:50] tsnag: Hello! [16:44:59] tsnag: Are you here? [16:49:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:50:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:53:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:54:19] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2016.000000 [16:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:59:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87358 MB (14% inode=99%): [16:59:19] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [17:05:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:08:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:12:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:28:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [17:32:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3621.000000 [17:41:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:49:13] is there any page about planned future of ts maintenance? (i don't mean discussion about future of ts, but stuff like: there will be apache installed on during october 2012, linux will replace solaris on during ... etc...) [17:50:24] outdated software will replace solaris? [17:50:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:51:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:53:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:54:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:59:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87184 MB (14% inode=99%): [17:59:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [18:05:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:08:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:12:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:18:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:28:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [18:32:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5341.000000 [18:42:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:50:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:51:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:52:16] what's up with SGE? jobs don't seem to be running? [18:53:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:59:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86943 MB (14% inode=99%): [18:59:19] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [19:05:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:06:15] Nettrom: which job? [19:08:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:11:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:13:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:28:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [19:32:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6717.000000 [19:43:03] DaBPunkt: I was trying to figure it out... was cronie on the submit-server down, perhaps? qstat didn't seem to report my hourly opentask-updater queued [19:43:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:45:58] noticed there was a gap in the updates here http://en.wikipedia.org/w/index.php?title=Wikipedia:Community_portal/Opentask&curid=399197&action=history [19:46:12] but couldn't find any info about why it stopped updating... qstat listed nothing [19:46:31] not a big deal since it seems to be running again now [19:48:13] DaBPunkt: fcgi (for tcl) seems a bit … dead, example: http://toolserver.org/~gifti/tool/namenstage.fcgi?char=A [19:48:13] unfortunately I have to get going [19:48:35] do you have an idea? [19:50:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:52:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:53:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:54:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:59:28] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [20:00:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86730 MB (14% inode=99%): [20:05:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:08:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:14:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:18:18] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1852 [20:18:28] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1838.000000 [20:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:19:28] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [20:20:18] MySQL slave on rosemary is OK: Uptime: 15138170 Threads: 28 Questions: 6719847375 Slow queries: 2136396 Opens: 298347 Flush tables: 6 Open tables: 3892 Queries per second avg: 443.900 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1734 [20:28:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [20:33:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8782.000000 [20:34:28] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/4:UP:1 UP: OK [20:34:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/2:UP:1 UP: OK [20:42:48] DaBPunkt: fyi: time to time /mnt/usr-store/dumps/... (various subdirs) on nightshade returns "can't read directory content" [20:43:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:45:04] Danny_B|backup: can you leave the user-store for a moment? I will try to re-mount it [20:45:35] let me change the dir in mc, otherwise i don't use it [20:46:04] done [20:46:10] which server? [20:46:37] nightshade. but only Danny_B|backup is using it AFAIS [20:47:37] okok [20:50:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:52:06] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:53:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:53:22] Danny_B|backup: there is still a screen by you that points to the user-store [20:53:37] let me check [20:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:54:41] should be off now [20:58:44] ok, try now [21:01:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86492 MB (14% inode=99%): [21:05:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:06:16] seems to response much faster now [21:08:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:11:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:14:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:16:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 104960 MB (10% inode=99%): [21:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:34:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11584.000000 [21:44:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:49:28] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73324 MB (7% inode=98%): [21:50:07] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37813 MB (9% inode=99%): [21:50:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:52:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:53:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:01:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86357 MB (14% inode=99%): [22:04:18] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73327 MB (7% inode=98%): [22:05:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:08:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:14:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:20:47] hmm ,it again behaves slower and slower [22:34:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14208.000000 [22:44:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:50:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:52:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:53:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:54:28] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:02:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86231 MB (14% inode=99%): [23:05:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:08:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:11:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:17:29] nacht ts [23:18:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:20:46] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1814.000000 [23:21:28] Load avg. on adenia is WARNING: WARNING - load average: 15.71, 14.73, 11.55 [23:24:46] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1795.000000 [23:28:27] Load avg. on adenia is OK: OK - load average: 9.82, 13.78, 12.48 [23:35:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15981.000000 [23:37:28] Load avg. on adenia is WARNING: WARNING - load average: 14.75, 15.05, 13.76 [23:44:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:50:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:52:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:53:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:54:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output