[00:01:46] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1894 [00:02:28] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.858398/1.75, alarm hl:np_load_avg=0.695312/2.00, alarm hl:mem_free=215.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.858398/1.50, alarm hl:np_load_long=0.562500/1.75, alarm hl:mem_free=215.000000M/250M [00:07:27] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:10:26] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.846680/1.75, alarm hl:np_load_avg=0.666992/2.00, alarm hl:mem_free=199.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.846680/1.50, alarm hl:np_load_long=0.594727/1.75, alarm hl:mem_free=199.000000M/250M [00:18:33] @replag [00:18:33] MC8: s1-sec: 8m 10s [+0.01 s/s]; s3-rr: 48m 37s [+0.10 s/s]; s3-user: 48m 37s [+0.10 s/s]; s5-rr: 10s [+0.00 s/s]; s5-user: 10s [-0.00 s/s]; s6-rr: 29m 16s [+0.03 s/s]; s6-user: 29m 16s [+0.03 s/s]; s7-rr: 32m 1s [+0.03 s/s] [00:18:34] MC8: s7-user: 32m 1s [+0.03 s/s] [00:18:46] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1918 [00:18:59] * MC8 blinks, thought there were only three, not seven [00:19:28] Some of them are sharing a machine. [00:19:44] I don't know how many clusters Wikimedia has at this point. [00:19:50] 5, maybe? [00:22:16] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1901 [00:23:16] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:23:56] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:32:47] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3644 [00:43:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:47:16] MySQL slave on z-dat-s6-a is OK: Uptime: 5796186 Threads: 14 Questions: 1115265620 Slow queries: 391131 Opens: 11415077 Flush tables: 2 Open tables: 2916 Queries per second avg: 192.413 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1748 [00:49:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47570 MB (4% inode=99%): [00:52:16] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:53:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:02:27] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.769531/1.75, alarm hl:np_load_avg=0.723144/2.00, alarm hl:mem_free=282.000000M/300M [01:03:27] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:06:28] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.668457/1.75, alarm hl:np_load_avg=0.679688/2.00, alarm hl:mem_free=241.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.668457/1.50, alarm hl:np_load_long=0.614746/1.75, alarm hl:mem_free=241.000000M/250M [01:08:38] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3633 [01:10:52] zzz =____= [01:21:57] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (FATAL: the database system is in recovery mode [01:29:58] PostgreSQL on ptolemy is OK: OK - database osm_mapnik (0 sec.) [01:31:12] [[Special:Log/newusers]] create 10 * NimronAdik * (New user account) [01:32:39] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3543 [01:32:48] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5449 [01:43:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:45:49] Betacommand: You got a second? [01:47:29] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.442383/1.75, alarm hl:np_load_avg=0.468262/2.00, alarm hl:mem_free=296.000000M/300M [01:49:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47373 MB (4% inode=99%): [01:49:27] TParis: yeah [01:49:29] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:51:10] Betacommand: How do I make a php ircscript run when I close the console? [01:51:10] [[User talk:NimronAdik]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6605&rcid=8699 * NimronAdik * (+2696) (Created page with "Just how long perhaps you have had your own carpet? Just when was the final time you'd this cleaned? It is important to know that the build-up of grime, grime and bacteria could ...") [01:51:27] TParis: use screen [01:51:34] example [01:51:46] screen php persist.php [01:51:54] thanks Betacommand, I knew you'd know [01:52:02] TParis: :) [01:52:04] And it'll show up in the proc-list when I do top? [01:52:10] So I can kill it if I need to? [01:52:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:52:45] More spam. [01:52:46] :-( [01:53:02] TParis: You should read the wiki page on scren. [01:53:11] https://wiki.toolserver.org/view/Screen [01:53:13] Alright, I will. Thanks. [01:53:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:53:41] [[Special:Log/move]] move 10 * NimronAdik * (moved [[02User talk:NimronAdik10]] to [[Carpet Cleaning Ipswich]]) [01:54:13] Joan: Makes perfect sense, thanks for pointing it out [01:56:28] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.685059/1.75, alarm hl:np_load_avg=0.590820/2.00, alarm hl:mem_free=244.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.685059/1.50, alarm hl:np_load_long=0.548828/1.75, alarm hl:mem_free=244.000000M/250M [01:57:04] [[Special:Log/delete]] delete 10 * Betacommand * (deleted "[[02Carpet Cleaning Ipswich10]]": Vandalism: content was: "Just how long perhaps you have had your own carpet? Just when was the final time you'd this cleaned? It is important to know that the build-up of grime, gri..." (and the only contributor was "[[Special:Contributions/NimronAdik|NimronAdik]]")) [01:57:16] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:NimronAdik10]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [01:57:25] Joan: gone [01:57:54] [[Special:Log/delete]] delete 10 * Betacommand * (deleted "[[02User talk:NimronAdik10]]": non-use: content was: "#REDIRECT [[Carpet Cleaning Ipswich]]" (and the only contributor was "[[Special:Contributions/NimronAdik|NimronAdik]]")) [02:13:30] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:21:30] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.615234/1.75, alarm hl:np_load_avg=0.595215/2.00, alarm hl:mem_free=289.000000M/300M [02:22:29] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:23:31] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.957031/1.00, alarm hl:np_load_long=0.850586/1.50, alarm hl:mem_free=23253.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.957031/1.10, alarm hl:np_load_long=0.850586/1.75, alarm hl:mem_free=23253.000000M/300M [02:25:31] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [02:32:39] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2693 [02:32:58] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7312 [02:44:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:49:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47226 MB (4% inode=99%): [02:52:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:53:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:12:08] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3627 [03:28:29] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.484375/1.00, alarm hl:np_load_long=0.634765/1.50, alarm hl:mem_free=23387.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.484375/1.10, alarm hl:np_load_long=0.634765/1.75, alarm hl:mem_free=23387.000000M/300M [03:29:30] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [03:31:21] 3(closed) [UTRS-5] IRC Feed of Requests <10https://jira.toolserver.org/browse/UTRS-5> (Andrew Pearson) [03:31:24] 3(resolved) [UTRS-27] Trying to release a closed bug gives a horrible error message <10https://jira.toolserver.org/browse/UTRS-27> (Andrew Pearson) [03:32:30] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.569824/1.75, alarm hl:np_load_avg=0.596191/2.00, alarm hl:mem_free=284.000000M/300M [03:33:08] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8607 [03:33:21] 3(closed) [UTRS-8] Add read-only API for IRC and On-wiki bots <10https://jira.toolserver.org/browse/UTRS-8> (Andrew Pearson) [03:34:30] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:35:21] 3(work started) [UTRS-24] Template.php the toolbar with links to "Appeal a block" and "Guide to appealing blocks" go to home.php for non-logged in users <10https://jira.toolserver.org/browse/UTRS-24> (Andrew Pearson) [03:35:24] 3(resolved) [UTRS-24] Template.php the toolbar with links to "Appeal a block" and "Guide to appealing blocks" go to home.php for non-logged in users <10https://jira.toolserver.org/browse/UTRS-24> (Andrew Pearson) [03:42:22] 3(work started) [UTRS-26] Add help for "Reserve", "Release" and "Close" buttons <10https://jira.toolserver.org/browse/UTRS-26> (Andrew Pearson) [03:44:30] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:47:21] 3(resolved) [UTRS-26] Add help for "Reserve", "Release" and "Close" buttons <10https://jira.toolserver.org/browse/UTRS-26> (Andrew Pearson) [03:49:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47086 MB (4% inode=99%): [03:52:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:53:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:12:49] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6572 [04:33:58] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8694 [04:44:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:44:50] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3456 [04:47:58] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2983 [04:48:50] MySQL slave on z-dat-s7-a is OK: Uptime: 3316193 Threads: 7 Questions: 789836357 Slow queries: 136661 Opens: 9296731 Flush tables: 1 Open tables: 3925 Queries per second avg: 238.175 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1312 [04:49:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46930 MB (4% inode=99%): [04:49:58] MySQL slave on z-dat-s3-a is OK: Uptime: 4543825 Threads: 13 Questions: 5168481674 Slow queries: 378468 Opens: 65263336 Flush tables: 2 Open tables: 16384 Queries per second avg: 1137.473 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1609 [04:53:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:53:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:02:28] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.645019/1.75, alarm hl:np_load_avg=0.652832/2.00, alarm hl:mem_free=230.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.645019/1.50, alarm hl:np_load_long=0.597168/1.75, alarm hl:mem_free=230.000000M/250M [05:07:29] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:13:30] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.639648/1.75, alarm hl:np_load_avg=0.585938/2.00, alarm hl:mem_free=279.000000M/300M [05:22:02] unitedcabletv * [Toolserver-l] (no subject) [05:44:40] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:49:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47749 MB (4% inode=99%): [05:52:43] [[Special:Log/newusers]] create 10 * Janicejohnson * (New user account) [05:53:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:53:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:06:28] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.635254/1.75, alarm hl:np_load_avg=0.755859/2.00, alarm hl:mem_free=205.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.635254/1.50, alarm hl:np_load_long=0.741211/1.75, alarm hl:mem_free=205.000000M/250M [06:08:28] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:12:30] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.644531/1.75, alarm hl:np_load_avg=0.662598/2.00, alarm hl:mem_free=216.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.644531/1.50, alarm hl:np_load_long=0.698242/1.75, alarm hl:mem_free=216.000000M/250M [06:44:36] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.134766/1.00, alarm hl:np_load_long=0.843750/1.50, alarm hl:mem_free=23392.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.134766/1.10, alarm hl:np_load_long=0.843750/1.75, alarm hl:mem_free=23392.000000M/300M [06:45:36] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [06:45:36] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:50:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47613 MB (4% inode=99%): [06:54:04] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (FATAL: the database system is in recovery mode [06:54:14] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:54:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:02:05] PostgreSQL on ptolemy is OK: OK - database osm_mapnik (0 sec.) [07:02:35] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.791016/1.75, alarm hl:np_load_avg=0.769531/2.00, alarm hl:mem_free=286.000000M/300M [07:05:35] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:15:10] [[Special:Log/newusers]] create 10 * Suzannephillips * (New user account) [07:46:36] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:50:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47465 MB (4% inode=99%): [07:54:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:54:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:07:20] 3(created) [WPONOSM-6] in-article map scale with dim=200; Wikipedia on OSM; Bug <10https://jira.toolserver.org/browse/WPONOSM-6> (Kentaur) [08:12:36] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.706055/1.75, alarm hl:np_load_avg=0.621094/2.00, alarm hl:mem_free=192.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.706055/1.50, alarm hl:np_load_long=0.579102/1.75, alarm hl:mem_free=192.000000M/250M [08:13:35] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:15:56] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 23653 MB (2% inode=99%): [08:46:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:50:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46857 MB (4% inode=99%): [08:54:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:55:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:57:45] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.512695/1.75, alarm hl:np_load_avg=0.568848/2.00, alarm hl:mem_free=150.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.512695/1.50, alarm hl:np_load_long=0.551758/1.75, alarm hl:mem_free=150.000000M/250M [09:14:46] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:17:26] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.106445/1.75, alarm hl:np_load_avg=0.140137/2.00, alarm hl:mem_free=222.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=0.106445/1.50, alarm hl:np_load_long=0.230469/1.75, alarm hl:mem_free=222.000000M/250M [09:18:25] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [09:20:46] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.780762/1.75, alarm hl:np_load_avg=0.576660/2.00, alarm hl:mem_free=144.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.780762/1.50, alarm hl:np_load_long=0.560547/1.75, alarm hl:mem_free=144.000000M/250M [09:22:25] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.146973/1.75, alarm hl:np_load_avg=0.142578/2.00, alarm hl:mem_free=219.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=0.146973/1.50, alarm hl:np_load_long=0.204590/1.75, alarm hl:mem_free=219.000000M/250M [09:22:46] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:42:45] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.069336/1.00, alarm hl:np_load_long=0.527344/1.50, alarm hl:mem_free=21966.000000M/300M [09:43:46] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [09:46:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:51:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47232 MB (4% inode=99%): [09:54:35] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:55:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:10:20] 3(commented) [DRTRIGON-112] subster_irc bot forgets wiki login when accessing other mediawiki project <10https://jira.toolserver.org/browse/DRTRIGON-112> (drtrigon) [10:23:15] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:24] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:23:35] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:05] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:24:05] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:24:15] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [10:28:15] PostgreSQL on ptolemy is CRITICAL: CRITICAL - database osm_mapnik (9 sec.) [10:29:05] PostgreSQL on ptolemy is OK: OK - database osm_mapnik (0 sec.) [10:31:25] 3(commented) [OSM-3] Ptolemy Postgres crashed twice <10https://jira.toolserver.org/browse/OSM-3> (Marlen Caemmerer) [10:46:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:51:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47103 MB (4% inode=99%): [10:54:45] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:55:14] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:00:35] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:01:34] SMTP on z-dat-s7-a is OK: SMTP OK - 0.003 sec. response time [11:08:31] [[Toolserver:Sandbox]] ! 10https://wiki.toolserver.org/w/index.php?diff=6608&oldid=6567&rcid=8706 * 95.56.196.212 * (+12) () [11:09:09] [[Toolserver:Sandbox]] ! 10https://wiki.toolserver.org/w/index.php?diff=6609&oldid=6608&rcid=8707 * 95.56.196.212 * (+17) () [11:16:15] hello. is there a tools for searches in edit summaries? [11:17:45] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:51:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46968 MB (4% inode=99%): [11:54:45] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:55:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:13:45] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.506348/1.75, alarm hl:np_load_avg=0.529785/2.00, alarm hl:mem_free=285.000000M/300M [12:14:46] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:17:46] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.573731/1.75, alarm hl:np_load_avg=0.527344/2.00, alarm hl:mem_free=273.000000M/300M [12:46:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:51:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46815 MB (4% inode=99%): [12:54:46] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:55:25] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:06:58] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (FATAL: the database system is in recovery mode [13:13:10] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.547363/1.75, alarm hl:np_load_avg=0.481445/2.00, alarm hl:mem_free=282.000000M/300M [13:13:59] PostgreSQL on ptolemy is OK: OK - database osm_mapnik (0 sec.) [13:14:10] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:47:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:51:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47580 MB (4% inode=99%): [13:55:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:55:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:15:23] 3(created) [MNT-1181] Create view for mrwikisource; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1181> (DaB.) [14:17:23] 3(resolved) [MNT-1181] Create view for mrwikisource <10https://jira.toolserver.org/browse/MNT-1181> (DaB.) [14:19:22] 3(created) [MNT-1182] Create view for pnbwiktionary; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1182> (DaB.) [14:21:20] 3(resolved) [MNT-1182] Create view for pnbwiktionary <10https://jira.toolserver.org/browse/MNT-1182> (DaB.) [14:24:20] 3(assigned) [TS-1272] Remove group members from unblock project <10https://jira.toolserver.org/browse/TS-1272> (DaB.) [14:26:27] 3(assigned) [TS-1269] Requesting new JIRA project for my tools <10https://jira.toolserver.org/browse/TS-1269> (DaB.) [14:26:28] 3(resolved) [TS-1272] Remove group members from unblock project <10https://jira.toolserver.org/browse/TS-1272> (DaB.) [14:29:20] 3(resolved) [TS-1269] Requesting new JIRA project for my tools <10https://jira.toolserver.org/browse/TS-1269> (DaB.) [14:31:22] 3(commented) [TS-1261] Create MMP "dumps" <10https://jira.toolserver.org/browse/TS-1261> (DaB.) [14:31:25] 3(assigned) [TS-1267] Install gcj on nightshade <10https://jira.toolserver.org/browse/TS-1267> (DaB.) [14:40:24] 3(resolved) [TS-1257] Toolserver wiki is pulling CC icon from (insecure) remote host <10https://jira.toolserver.org/browse/TS-1257> (DaB.) [14:44:21] 3(resolved) [TS-1161] Need Date::Calc <10https://jira.toolserver.org/browse/TS-1161> (DaB.) [14:47:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:51:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47411 MB (4% inode=99%): [14:55:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:55:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:02:17] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.412109/1.75, alarm hl:np_load_avg=0.449219/2.00, alarm hl:mem_free=287.000000M/300M [15:03:17] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:06:18] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.418457/1.75, alarm hl:np_load_avg=0.420410/2.00, alarm hl:mem_free=276.000000M/300M [15:34:22] 3(created) [ACCAPP-445] Statistics summarizer for Wikimedia Deutschland; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-445> (Kai Nissen) [15:47:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:52:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47229 MB (4% inode=99%): [15:55:17] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:56:57] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:02:17] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.605957/1.75, alarm hl:np_load_avg=0.560059/2.00, alarm hl:mem_free=290.000000M/300M [16:03:18] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [16:29:17] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.653320/1.00, alarm hl:np_load_long=0.741211/1.50, alarm hl:mem_free=23492.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.653320/1.10, alarm hl:np_load_long=0.741211/1.75, alarm hl:mem_free=23492.000000M/300M [16:30:18] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [16:47:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:52:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46991 MB (4% inode=99%): [16:55:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:56:17] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.528320/1.75, alarm hl:np_load_avg=0.528320/2.00, alarm hl:mem_free=217.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.528320/1.50, alarm hl:np_load_long=0.498047/1.75, alarm hl:mem_free=217.000000M/250M [16:56:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:59:18] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [17:02:18] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.635742/1.75, alarm hl:np_load_avg=0.590332/2.00, alarm hl:mem_free=298.000000M/300M [17:37:24] 3(created) [OSM-5] hstore operator ? no longer uses indexes; OSM; Bug <10https://jira.toolserver.org/browse/OSM-5> (Kai Krueger) [17:47:26] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:52:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46826 MB (4% inode=99%): [17:55:27] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:56:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:25:18] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.058594/1.00, alarm hl:np_load_long=0.757812/1.50, alarm hl:mem_free=23165.000000M/300M [18:47:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:53:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47483 MB (4% inode=99%): [18:55:27] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:57:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:07:27] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.421387/1.75, alarm hl:np_load_avg=0.412109/2.00, alarm hl:mem_free=239.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.421387/1.50, alarm hl:np_load_long=0.384277/1.75, alarm hl:mem_free=239.000000M/250M [19:08:27] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [19:09:27] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [19:33:28] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.829102/1.00, alarm hl:np_load_long=1.289062/1.50, alarm hl:mem_free=22090.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.829102/1.10, alarm hl:np_load_long=1.289062/1.75, alarm hl:mem_free=22090.000000M/300M [19:36:27] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [19:47:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:53:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47303 MB (4% inode=99%): [19:55:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:58:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:59:02] DaB. * [Toolserver-announce] Regular Maintenance at 8. February [19:59:21] finaly! That was the 3. try [20:01:09] mailinglist discarding your mails? :-) [20:02:28] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.736328/1.75, alarm hl:np_load_avg=0.626465/2.00, alarm hl:mem_free=281.000000M/300M [20:02:53] valhallasw: yes [20:04:27] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [20:07:27] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.625000/1.75, alarm hl:np_load_avg=0.624512/2.00, alarm hl:mem_free=234.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.625000/1.50, alarm hl:np_load_long=0.508301/1.75, alarm hl:mem_free=234.000000M/250M [20:13:26] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.189453/1.00, alarm hl:np_load_long=0.937500/1.50, alarm hl:mem_free=22426.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.189453/1.10, alarm hl:np_load_long=0.937500/1.75, alarm hl:mem_free=22426.000000M/300M [20:16:27] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 22467 MB (2% inode=99%): [20:17:28] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [20:37:20] 3(resolved) [WPONOSM-6] in-article map scale with dim=200 <10https://jira.toolserver.org/browse/WPONOSM-6> (Kolossos ) [20:37:29] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.761719/1.00, alarm hl:np_load_long=0.990235/1.50, alarm hl:mem_free=22121.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.761719/1.10, alarm hl:np_load_long=0.990235/1.75, alarm hl:mem_free=22121.000000M/300M [20:40:28] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [20:47:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:53:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47124 MB (4% inode=99%): [20:55:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:58:07] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:14:28] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.156250/1.00, alarm hl:np_load_long=0.915039/1.50, alarm hl:mem_free=21767.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.156250/1.10, alarm hl:np_load_long=0.915039/1.75, alarm hl:mem_free=21767.000000M/300M [21:32:07] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:08] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:32:28] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [21:32:46] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [21:32:57] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:34:20] 3(created) [MATTHEWRBOWKER-1] Add functionality for !link; Matthewrbowker's Tools: WikiWelcomer; New Feature <10https://jira.toolserver.org/browse/MATTHEWRBOWKER-1> (Matthew Bowker) [21:35:29] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.174805/1.00, alarm hl:np_load_long=1.125000/1.50, alarm hl:mem_free=21921.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.174805/1.10, alarm hl:np_load_long=1.125000/1.75, alarm hl:mem_free=21921.000000M/300M [21:44:20] 3(commented) [MATTHEWRBOWKER-1] Add functionality for !link <10https://jira.toolserver.org/browse/MATTHEWRBOWKER-1> (Matthew Bowker) [21:47:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:53:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46972 MB (4% inode=99%): [21:55:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:58:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:03:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.309570/1.75, alarm hl:np_load_avg=0.360840/2.00, alarm hl:mem_free=282.000000M/300M [22:07:37] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [22:12:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.445801/1.75, alarm hl:np_load_avg=0.399902/2.00, alarm hl:mem_free=143.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.445801/1.50, alarm hl:np_load_long=0.413574/1.75, alarm hl:mem_free=143.000000M/250M [22:32:39] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [22:47:39] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:53:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46820 MB (4% inode=99%): [22:55:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:57:39] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.318848/1.75, alarm hl:np_load_avg=0.376465/2.00, alarm hl:mem_free=214.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.318848/1.50, alarm hl:np_load_long=0.369141/1.75, alarm hl:mem_free=214.000000M/250M [22:58:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:03:38] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:06:39] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.422363/1.75, alarm hl:np_load_avg=0.398438/2.00, alarm hl:mem_free=221.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.422363/1.50, alarm hl:np_load_long=0.383789/1.75, alarm hl:mem_free=221.000000M/250M [23:26:40] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [23:48:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:54:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46658 MB (4% inode=99%): [23:56:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:58:17] SSH on nightshade.mgmt is CRITICAL: Server answer: