[00:01:05] [[Category:Tools by authors]] ! 10https://wiki.toolserver.org/w/index.php?diff=6617&oldid=52&rcid=8720 * 68.44.245.240 * (+2330) (Life History of Dr. Joachim Ifezuo Oforchukwu, Ph.D.) [00:02:19] story of my life [00:02:35] [[Category:Tools by authors]] M 10https://wiki.toolserver.org/w/index.php?diff=6618&oldid=6617&rcid=8721 * Valhallasw * (-2330) (Reverted edits by [[Special:Contributions/68.44.245.240|68.44.245.240]] ([[User talk:68.44.245.240|talk]]) to last revision by [[User:Agony|Agony]]) [00:03:01] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.201172/1.75, alarm hl:np_load_avg=1.119141/2.00, alarm hl:mem_free=227.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.201172/1.50, alarm hl:np_load_long=0.885254/1.75, alarm hl:mem_free=227.000000M/250M [00:03:02] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.711426/1.50, alarm hl:np_load_long=1.147949/1.75, alarm hl:mem_free=386.000000M/250M [00:03:07] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:68.44.245.24010]] with an expiry time of infinite (anonymous users only, account creation disabled): Inserting nonsense/gibberish into pages) [00:04:01] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [00:05:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45771 MB (4% inode=99%): [00:09:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:10:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:13:00] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.928711/1.75, alarm hl:np_load_avg=1.078613/2.00, alarm hl:mem_free=274.000000M/300M [00:29:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:33:01] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.826660/1.75, alarm hl:np_load_avg=0.886231/2.00, alarm hl:mem_free=180.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.826660/1.50, alarm hl:np_load_long=0.921387/1.75, alarm hl:mem_free=180.000000M/250M [00:35:12] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:59:10] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:05:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45737 MB (4% inode=99%): [01:09:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:10:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:33:10] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.926758/1.75, alarm hl:np_load_avg=1.019531/2.00, alarm hl:mem_free=268.000000M/300M [01:35:10] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [01:39:12] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.926758/1.75, alarm hl:np_load_avg=0.901367/2.00, alarm hl:mem_free=290.000000M/300M [01:49:08] [[Recent moves]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6619&rcid=8723 * Dcoetzee * (+758) (Create) [01:59:09] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:05:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45628 MB (4% inode=99%): [02:09:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:10:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:20:20] 3(commented) [DRTRIGON-112] subster_irc bot forgets wiki login when accessing other mediawiki project <10https://jira.toolserver.org/browse/DRTRIGON-112> (drtrigon) [02:29:11] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.842773/1.75, alarm hl:np_load_avg=0.838379/2.00, alarm hl:mem_free=258.000000M/300M [02:30:10] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [02:42:12] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.944824/1.75, alarm hl:np_load_avg=0.874023/2.00, alarm hl:mem_free=197.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=0.944824/1.50, alarm hl:np_load_long=0.868652/1.75, alarm hl:mem_free=197.000000M/250M [02:52:32] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:50] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:50] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:00] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:53:00] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:10] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:53:21] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:30] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:53:40] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:41] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:50] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [02:54:31] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [02:54:40] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:50] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3220 MB (99% inode=99%): [02:54:50] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:54:50] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:54:50] MySQL on z-dat-s3-a is OK: Uptime: 4796122 Threads: 15 Questions: 5543525523 Slow queries: 392477 Opens: 68764094 Flush tables: 2 Open tables: 16384 Queries per second avg: 1155.834 [02:55:00] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3432 MB (99% inode=99%): [02:55:11] MySQL slave on z-dat-s3-a is OK: Uptime: 4796135 Threads: 11 Questions: 5543540915 Slow queries: 392479 Opens: 68764147 Flush tables: 2 Open tables: 16384 Queries per second avg: 1155.835 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 232 [02:55:11] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:55:22] SMTP on hyacinth is OK: SMTP OK - 0.076 sec. response time [02:55:31] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [02:55:31] SMTP on z-dat-s3-a is OK: SMTP OK - 0.003 sec. response time [02:55:31] SMTP on z-dat-s7-a is OK: SMTP OK - 0.173 sec. response time [02:55:40] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:55:40] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:59:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:05:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45489 MB (4% inode=99%): [03:09:10] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:10:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:13:11] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.715332/1.75, alarm hl:np_load_avg=0.633301/2.00, alarm hl:mem_free=283.000000M/300M [03:14:21] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.030274/1.00, alarm hl:np_load_long=0.765625/1.50, alarm hl:mem_free=22305.000000M/300M [03:15:21] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [03:17:10] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:22:51] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:00] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:23:01] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:01] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:40] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:40] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:23:40] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:23:40] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:23:51] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:23:51] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:24:12] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3565 MB (99% inode=99%): [03:24:12] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.858398/1.75, alarm hl:np_load_avg=0.876953/2.00, alarm hl:mem_free=272.000000M/300M [03:24:31] SMTP on z-dat-s6-a is OK: SMTP OK - 0.002 sec. response time [03:25:13] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [03:46:20] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.026367/1.00, alarm hl:np_load_long=0.875000/1.50, alarm hl:mem_free=22016.000000M/300M [03:48:20] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [03:59:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:01:40] zzz =_= [04:05:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46219 MB (4% inode=99%): [04:09:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:10:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:20:21] 3(commented) [DBQ-174] Emijrp/List of Wikipedians by number of edits <10https://jira.toolserver.org/browse/DBQ-174> (Rahuldeshmukh101) [04:59:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:03:31] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.477539/1.00, alarm hl:np_load_long=0.875977/1.50, alarm hl:mem_free=21703.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.477539/1.10, alarm hl:np_load_long=0.875977/1.75, alarm hl:mem_free=21703.000000M/300M [05:05:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45941 MB (4% inode=99%): [05:05:31] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [05:10:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:10:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:23:40] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.020508/1.00, alarm hl:np_load_long=0.833985/1.50, alarm hl:mem_free=21741.000000M/300M [05:59:36] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:05:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45502 MB (4% inode=99%): [06:10:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:10:36] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:23:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:24:03] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [06:28:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:40:04] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:44] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:04] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:04] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:04] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:04] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:04] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [06:41:04] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:41:04] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:41:05] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:41:13] MySQL on z-dat-s3-a is OK: Uptime: 4809702 Threads: 20 Questions: 5561347095 Slow queries: 393480 Opens: 68947127 Flush tables: 2 Open tables: 16384 Queries per second avg: 1156.276 [06:41:35] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 121532 MB (30% inode=99%): [06:41:35] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3483 MB (99% inode=99%): [06:41:35] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3466 MB (99% inode=99%): [06:41:35] SMTP on hyacinth is OK: SMTP OK - 0.119 sec. response time [06:41:55] SMTP on z-dat-s7-a is OK: SMTP OK - 0.022 sec. response time [06:41:57] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:41:57] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:41:57] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:41:57] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [07:00:36] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:05:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45345 MB (4% inode=99%): [07:11:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:11:35] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:00:44] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:06:35] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45103 MB (4% inode=99%): [08:11:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:11:36] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:17:54] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 19953 MB (2% inode=99%): [08:31:05] [[Talk:User-store]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6620&rcid=8724 * Nemobis * (+8278) (update?) [08:33:36] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.486328/1.75, alarm hl:np_load_avg=1.101074/2.00, alarm hl:mem_free=298.000000M/300M [08:38:34] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [09:00:44] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:07:35] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45893 MB (4% inode=99%): [09:11:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:11:36] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:31:03] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.127930/1.00, alarm hl:np_load_long=0.924805/1.50, alarm hl:mem_free=23274.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.127930/1.10, alarm hl:np_load_long=0.924805/1.75, alarm hl:mem_free=23274.000000M/300M [09:32:04] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [10:00:44] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:08:35] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45770 MB (4% inode=99%): [10:12:34] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:12:34] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:29:36] [[Tool considerations]] ! 10https://wiki.toolserver.org/w/index.php?diff=6621&oldid=5933&rcid=8725 * 176.241.32.164 * (-7) (/* Security */ too much) [10:39:45] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.719238/1.75, alarm hl:np_load_avg=0.662598/2.00, alarm hl:mem_free=172.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=0.719238/1.50, alarm hl:np_load_long=0.681641/1.75, alarm hl:mem_free=172.000000M/250M [10:40:46] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [10:52:33] hi [10:53:22] i just saw that some hours ago "/mnt/user-store/dump/itwiki-20120109-pages-meta-current.xml" has been deleted... but i need to work with it. how? [10:53:55] download it again? [10:54:18] or maybe it has been moved [10:54:55] hm... i don't think to have rights to download in /mnt/user-store/dump/ and for sure before redownload, i need to know why it was deleted [10:55:27] dumps are being consolidated now [10:55:58] to be located only on one place and in some systematic naming [10:56:12] that sounds great [10:56:40] danny_b|backup so where will i find itwiki dumps? [10:56:49] ps. that's great! [10:57:34] i saw that there is a /mnt/user-store/dumps/itwiki [10:57:35] nickanc: there will be post in list about the new structure. current locations will be kept for a little while via symlinks [10:57:50] but it contains only articles, not meta-current dump [10:58:23] ok [10:59:04] so, for now, no meta-current dump for itwiki? [10:59:28] * nickanc waves to valhallasw :) [10:59:28] the entire usr-store is one big mess which needs systematic approach [11:00:11] nickanc: i was not doing anything with itwiki dumps [11:00:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:00:50] oh [11:01:18] i was looking for what yesterday was located at /mnt/user-store/dump/itwiki-20120109-pages-meta-current.xml because i don't find it [11:01:57] i assume sk was doing something with it [11:02:33] damn, i would desperately need higher rights to perform that maintenance [11:02:47] can't move some things [11:03:31] it would be nice if we could just mirror the latest files from dumps.wikimedia.org. [11:04:04] * nickanc thinks to solve his problem with -start:! [11:04:06] johang: that's in plan [11:04:33] johang: but first the cleanup is needed [11:05:21] indeed. [11:05:36] Danny_B|backup: are you admin or something on TS? [11:06:03] nope, just have a little higher rights for dump dirs to perform the maintenance [11:06:27] that's why i was saying i'd need higher rights to perform the entire maintenance [11:06:45] right. [11:07:47] why the hack *xml* file or *bz2* file are rwX??? [11:08:37] just r is enough, I'd say [11:08:47] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45565 MB (4% inode=99%): [11:09:05] johang Danny i think there was something about it in the list+ [11:09:15] *list [11:09:31] yes, sk wrote it [11:09:39] but i don't see any reason to have X [11:09:42] I'd love to see larger /tmp directories on toolserver, speaking on maintenance. [11:09:54] rw-rw-rw- is good enough [11:10:14] johang: -> DaBPunkt or nosy [11:11:32] not online atm :| [11:12:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:12:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:23:14] johang send a memo to them [11:26:44] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=0.667481/1.75, alarm hl:np_load_avg=0.653809/2.00, alarm hl:mem_free=140.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=0.667481/1.50, alarm hl:np_load_long=0.673828/1.75, alarm hl:mem_free=140.000000M/250M [11:27:45] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [12:00:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:08:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45451 MB (4% inode=99%): [12:12:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:12:46] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:00:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:04:38] johang: if you submit a task to sge with e.g. -l tmp_free=100M, sge creates a local tmp-dir for you which will have the space for sure. You only have to use $TMP and not /tmp [13:05:20] 3(created) [UTRS-48] Apostrophe shows with \ on "Why do you believe you should be unblocked?"; UTRS; Minor Bug <10https://jira.toolserver.org/browse/UTRS-48> (Thehelpfulone) [13:09:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45323 MB (4% inode=99%): [13:12:44] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.635254/1.75, alarm hl:np_load_avg=0.583984/2.00, alarm hl:mem_free=135.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.635254/1.50, alarm hl:np_load_long=0.589356/1.75, alarm hl:mem_free=135.000000M/250M [13:13:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:13:45] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:13:45] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:37:21] 3(commented) [UTRS-47] Notification of reply <10https://jira.toolserver.org/browse/UTRS-47> (Thehelpfulone) [13:53:57] can group be owner? or only user? [14:00:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:01:45] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.706055/1.75, alarm hl:np_load_avg=0.546387/2.00, alarm hl:mem_free=162.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.706055/1.50, alarm hl:np_load_long=0.492188/1.75, alarm hl:mem_free=162.000000M/250M [14:02:54] Sun Grid Engine execd on wolfsbane is CRITICAL: short@wolfsbane in error state: QERROR as result of job 1554829s failure [14:06:54] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [14:09:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45162 MB (4% inode=99%): [14:13:04] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.602539/1.00, alarm hl:np_load_long=0.834961/1.50, alarm hl:mem_free=22208.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.602539/1.10, alarm hl:np_load_long=0.834961/1.75, alarm hl:mem_free=22208.000000M/300M [14:13:54] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:14:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:15:26] 3(resolved) [UTRS-48] Apostrophe shows with \ on "Why do you believe you should be unblocked?" <10https://jira.toolserver.org/browse/UTRS-48> (Andrew Pearson) [14:15:26] 3(assigned) [UTRS-48] Apostrophe shows with \ on "Why do you believe you should be unblocked?" <10https://jira.toolserver.org/browse/UTRS-48> (Andrew Pearson) [14:16:54] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.638672/1.75, alarm hl:np_load_avg=0.603027/2.00, alarm hl:mem_free=215.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.638672/1.50, alarm hl:np_load_long=0.541992/1.75, alarm hl:mem_free=215.000000M/250M [14:17:04] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [14:20:27] reflinks is not working. is it a toolserver problem? [14:22:29] mabdul|busy: url? [14:26:18] mmh, ocaasi can run it - seems a geolocation problem [14:26:40] https://toolserver.org/~dispenser/cgi-bin/webreflinks.py?page=Jairo_Barrull_Fern%C3%A1ndez&citeweb=on&overwrite=simple&limit=200 [14:29:16] mabdul|busy: tell dispenser dispenser@toolserver.org should work [14:30:40] DaBPunkt: mmh, mein problem is doch schon behoben XD [14:30:49] und irgendwo auf der welt funktionert der server ;) [14:45:55] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=1.055664/1.75, alarm hl:np_load_avg=0.801758/2.00, alarm hl:mem_free=223.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.055664/1.50, alarm hl:np_load_long=0.738769/1.75, alarm hl:mem_free=223.000000M/250M [14:46:54] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [15:01:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:02:54] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.492188/1.75, alarm hl:np_load_avg=0.552734/2.00, alarm hl:mem_free=295.000000M/300M [15:03:54] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:03:55] Sun Grid Engine execd on wolfsbane is CRITICAL: short@wolfsbane in error state: QERROR as result of job 1554829s failure [15:10:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45010 MB (4% inode=99%): [15:13:56] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:14:54] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:44:34] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [15:45:34] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1756.000000 [16:01:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:03:55] Sun Grid Engine execd on wolfsbane is CRITICAL: short@wolfsbane in error state: QERROR as result of job 1554829s failure [16:06:39] [[Special:Log/newusers]] create 10 * संतोष दहिवळ * (New user account) [16:10:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45802 MB (4% inode=99%): [16:14:55] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:14:55] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:38:21] 3(created) [ACCAPP-447] To run a Bot for scheduled work and routine event driven tasks on Marathi Wiipedia (mr.wiki).; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-447> (Rahuldeshmukh101) [16:56:55] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.624512/1.75, alarm hl:np_load_avg=0.573731/2.00, alarm hl:mem_free=163.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.624512/1.50, alarm hl:np_load_long=0.539062/1.75, alarm hl:mem_free=163.000000M/250M [16:58:55] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [17:01:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:03:55] Sun Grid Engine execd on wolfsbane is CRITICAL: short@wolfsbane in error state: QERROR as result of job 1554829s failure [17:10:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45624 MB (4% inode=99%): [17:13:13] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.024414/1.00, alarm hl:np_load_long=0.790039/1.50, alarm hl:mem_free=22218.000000M/300M [17:14:14] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:14:55] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:14:55] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:33:14] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.069336/1.00, alarm hl:np_load_long=0.827148/1.50, alarm hl:mem_free=22007.000000M/300M [17:49:22] 3(created) [ET-45] February newsletter delivery; JamesR's Tools; Minor Task <10https://jira.toolserver.org/browse/ET-45> (Keith Dorey) [18:01:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:04:04] Sun Grid Engine execd on wolfsbane is CRITICAL: short@wolfsbane in error state: QERROR as result of job 1554829s failure [18:10:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45472 MB (4% inode=99%): [18:14:54] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:14:54] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:28:59] hi [18:29:05] DaBPunkt you around? :) [18:29:22] yes [18:29:37] I was wondering if you could run your script again for deleted content on commons [18:29:54] I still have the numbers from september but want to update the statistics I hace [18:29:57] *have [18:30:01] DaBPunkt: is the ts-admins at wikimedia.org still valid? [18:30:19] it isnt urgent so it can wait if you have more important issues to deal with [18:30:21] Danny_B|backup: yes. I got your mail. I will respond later [18:31:02] ToAruShiroiNeko: ask me again in a few hours or tomorrow [18:31:28] I could also leave a message on your talk page :) [18:31:53] would that be fine? [18:32:05] no, please not. I would get organge-boxes on douzend of pages tomorow [18:32:20] ok :) [18:35:32] DaBPunkt: thx. i was just not sure if it's still valid [18:45:04] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.587402/1.75, alarm hl:np_load_avg=0.558594/2.00, alarm hl:mem_free=298.000000M/300M [18:49:04] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [18:53:03] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.572266/1.75, alarm hl:np_load_avg=0.617676/2.00, alarm hl:mem_free=282.000000M/300M [19:01:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:04:16] Sun Grid Engine execd on wolfsbane is CRITICAL: short@wolfsbane in error state: QERROR as result of job 1554829s failure [19:10:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45252 MB (4% inode=99%): [19:15:54] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:15:54] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:43:16] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.164062/1.00, alarm hl:np_load_long=0.749023/1.50, alarm hl:mem_free=22618.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.164062/1.10, alarm hl:np_load_long=0.749023/1.75, alarm hl:mem_free=22618.000000M/300M [19:43:19] zzz =_= [19:44:16] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [20:01:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:04:15] Sun Grid Engine execd on wolfsbane is CRITICAL: short@wolfsbane in error state: QERROR as result of job 1554829s failure [20:08:27] DaBPunkt less busy? :o [20:08:54] Merlissimo: thanks. have you noticed performance issues with my jobs? I want to fix it if that's the case. [20:10:21] johang: i only noticed that your jobs are causing many read/write blocks on nfs and could be much faster because most of the time they are waiting for i/o [20:11:14] Sun Grid Engine execd on wolfsbane is OK: short@wolfsbane OK: all.q@wolfsbane OK [20:11:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45081 MB (4% inode=99%): [20:12:39] while your jobs are running nagios is always reporting high load although many cpus are waiting [20:14:11] I'll investigate this. I read, process and write back logs to user-store so that's probably a correct observation. [20:15:11] maybe if I don't run so many jobs in parallel it might reduce the load. [20:15:25] you are reading from nfs user-store and piping this to a some scripts and writing the ouput to a temp-file also on user-store. [20:15:54] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:15:54] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:16:09] if you would create the temp-file on $tmp instead of user-store nfs would be much more efficient [20:16:12] the temp file is X.tmp, but the suffix is removed when the job finishes [20:16:47] the final destination is user-store, I'm just calling it .tmp so it can handle failures better. [20:17:07] just create it on the tmpdir and move it the user-store later. so nfs can do bulk read/write [20:17:20] ah, right. I get it. that makes sense. [20:18:15] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 20154 MB (2% inode=99%): [20:18:48] you can request die resource tmp_free (e.g. -l tmp_free=200M ) and sge creates a tmp-dir where you'll have the space for your file [20:19:18] just use $TMP and not /tmp [20:20:10] is that limit per job or per task? [20:20:35] for what? available tmp space? [20:20:39] yes [20:20:45] if I have an array of 5 tasks that all require, say, 100M, should I set it to 500M? [20:20:54] sge job array that is [20:21:12] it depends on the available space on each host. not limit per job [20:22:14] if you are running array jobs the value counts for each task. so -t 1-4 -l tmp_free=10M reserves 10M for every task [20:22:34] and not for the job in sum [20:23:35] very good. I'm starting to like SGE :) [20:26:03] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:26:24] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:24] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:46] I should make sort use $TMP too. [20:26:54] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:26:54] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:27:04] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:04] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:04] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:15] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:27:15] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:27:15] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:27:15] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:15] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:24] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:24] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:24] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:34] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [20:27:34] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [20:27:34] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [20:27:44] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [20:27:44] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [20:27:45] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [20:27:54] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [20:27:55] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [20:28:11] Merlissimo: what the rules on the different SGE queuess? when should I use short, longrun and all.q? [20:28:40] you should not select a special queue [20:28:57] sge will handle this, but you should define a maximum runtime [20:29:54] "-l h_rt=3:00:00" will set a maxumim runtime of 3 hours. [20:30:17] is that per array task or per job? [20:30:17] looks like hyacinth is down [20:30:20] 3(commented) [UTRS-19] Add list of inactive users for account disabling (suspension after x days) <10https://jira.toolserver.org/browse/UTRS-19> (AGK) [20:31:49] if you submit array tasks all values count always for a single task. [20:32:06] "SUL status [20:32:06] [20:32:06] can't connect to centralauth database." [20:32:52] per http://toolserver.org/~luxo/contributions/contributions.php?user=Jeff+G.&lang= [20:33:17] @replag [20:33:19] techman224: s3-rr: error; s3-user: error; s6-rr: error; s6-user: error; s7-rr: error; s7-user: error [20:33:33] guys, I need a moment to write a ticket :) [20:34:08] hyacinth's console is also down [20:34:23] I will hardreboot it [20:34:39] I'm only getting de and nl wikipedias [20:39:04] /tmp on hyacinth is CRITICAL: Connection refused by host [20:39:15] /v/sql on hyacinth is CRITICAL: Connection refused by host [20:39:15] Load avg. on hyacinth is CRITICAL: Connection refused by host [20:39:34] Environment on hyacinth is CRITICAL: Connection refused by host [20:42:20] 3(created) [UTRS-49] Reservation is until assignee navigates away from ticket, not for a specified time; UTRS; Improvement <10https://jira.toolserver.org/browse/UTRS-49> (AGK) [20:48:34] hyacinth's local-filesystem is not mounting at the moment. I look for the problem [20:49:45] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:49:45] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 55970 MB (99% inode=99%): [20:49:45] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [20:49:45] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 121614 MB (30% inode=99%): [20:49:54] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 44011 MB (99% inode=99%): [20:49:54] SMTP on z-dat-s4-a is OK: SMTP OK - 0.159 sec. response time [20:49:54] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:49:54] SMTP on hyacinth is OK: SMTP OK - 0.118 sec. response time [20:50:04] /tmp on hyacinth is OK: DISK OK - free space: /tmp 43827 MB (99% inode=99%): [20:50:04] SMTP on z-dat-s7-a is OK: SMTP OK - 0.194 sec. response time [20:50:04] SMTP on z-dat-s6-a is OK: SMTP OK - 0.239 sec. response time [20:50:14] SMTP on z-dat-s3-a is OK: SMTP OK - 0.128 sec. response time [20:50:14] /v/sql on hyacinth is OK: DISK OK - free space: /v/sql 203827 MB (21% inode=99%): [20:50:14] Load avg. on hyacinth is OK: OK - load average: 1.36, 0.68, 0.34 [20:50:14] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:50:14] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:50:24] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:50:25] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:50:25] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 43830 MB (99% inode=99%): [20:50:25] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 43830 MB (99% inode=99%): [20:51:56] found the problem. Looks like the sql-partion for s4 is away [20:54:54] MySQL slave on z-dat-s7-a is OK: Uptime: 303 Threads: 3 Questions: 10998 Slow queries: 1 Opens: 282 Flush tables: 1 Open tables: 269 Queries per second avg: 36.297 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1789 [20:54:54] MySQL on z-dat-s7-a is OK: Uptime: 304 Threads: 3 Questions: 11028 Slow queries: 1 Opens: 282 Flush tables: 1 Open tables: 269 Queries per second avg: 36.276 [20:55:40] halted the zone for s4 on hyacith, no need to let it consume memory [20:55:55] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1946 [20:56:15] MySQL on z-dat-s6-a is OK: Uptime: 383 Threads: 6 Questions: 2045 Slow queries: 0 Opens: 126 Flush tables: 1 Open tables: 115 Queries per second avg: 5.339 [20:56:54] MySQL slave on z-dat-s6-a is OK: Uptime: 423 Threads: 4 Questions: 9660 Slow queries: 1 Opens: 156 Flush tables: 1 Open tables: 145 Queries per second avg: 22.836 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [20:59:31] DaBPunkt: Is there a possibility to get access to DTrace profiling data? [21:00:53] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [21:01:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:03:17] apmon: I have no idea. Debugging is not my world [21:03:22] 3(created) [MNT-1183] Hyacinth crashed; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1183> (DaB.) [21:03:50] @replag [21:03:50] DaBPunkt: s1-pri: 45s [+0.00 s/s]; s1-sec: 41s [+0.00 s/s]; s3-rr: error; s3-user: error; s6-rr: 29m 35s [+0.01 s/s]; s6-user: 29m 35s [+0.01 s/s] [21:04:57] s3 is still running restore [21:05:19] One problem is that when I try things, I get permission denied. You presumably need to be an admin. Which is sort of understandable given that system level debugging would potentially be a privacy/securtiy risk [21:06:54] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2593 [21:06:54] MySQL on z-dat-s3-a is OK: Uptime: 1023 Threads: 18 Questions: 11798 Slow queries: 1 Opens: 392 Flush tables: 1 Open tables: 289 Queries per second avg: 11.532 [21:07:19] DaBPunkt: Do you know if nosy would be a better person to talk to about this? [21:07:22] apmon: speak with nosy, when she is back please [21:07:35] OK, will do [21:07:41] thanks [21:11:54] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [21:11:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44912 MB (4% inode=99%): [21:16:04] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:16:04] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:31:47] DaBPunkt back? :o [21:38:37] ToAruShiroiNeko: what is the problem? [21:54:32] hi [21:54:40] ok I wanted the statistics you provided before :) [21:54:53] on september you saved my skin by providing me with breakdown of files on commons [21:54:59] how many jpegs and etc [21:56:16] do you have the query I used? [21:56:55] hmm [21:57:14] let me look that up [21:57:25] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.262695/1.00, alarm hl:np_load_long=0.875000/1.50, alarm hl:mem_free=20772.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.262695/1.10, alarm hl:np_load_long=0.875000/1.75, alarm hl:mem_free=20772.000000M/300M [21:59:24] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [22:00:54] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [22:01:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:01:52] I cant seem to find it [22:02:08] ok, then please tell me what you need [22:02:13] http://wikimania2012.wikimedia.org/wiki/Submissions/CLEF_2011_-_Semi-automated_Artificial_Intelligence_to_assist_editing:_An_opportunity_for_Wikimedia_sites#Artificial_Intelligence [22:02:19] I want to update the breakdown there [22:03:09] I basically seek the files on commons based on their filetype [22:03:16] also I am curious what that lone mp4 is :/ [22:08:32] DaBPunkt I believe you had my request in the form of a text file on your toolserver account [22:08:38] which should have the sql query [22:11:54] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [22:11:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44745 MB (4% inode=99%): [22:12:51] ToAruShiroiNeko: http://toolserver.org/~dab/queries/ToAruShiroiNeko.2.txt this one? [22:16:04] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:16:04] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:28:43] DaBPunkt I believe so, yes [22:34:20] ToAruShiroiNeko: running [22:35:23] thanks [22:35:42] could you also run something that determines statistics on the number of deleted articles on en.wikipedia? [22:41:05] not tonight [22:49:00] @replag [22:49:01] DaBPunkt: s3-rr: 2h 25m 4s [+0.05 s/s]; s3-user: 2h 25m 5s [+0.05 s/s]; s6-rr: 2h 14m 46s [+1.00 s/s]; s6-user: 2h 14m 46s [+1.00 s/s] [23:00:54] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [23:02:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:11:54] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [23:12:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45538 MB (4% inode=99%): [23:16:05] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:16:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:26:21] 3(commented) [MNT-1183] Hyacinth crashed <10https://jira.toolserver.org/browse/MNT-1183> (DaB.) [23:38:54] @replag [23:38:56] DaBPunkt: s1-sec: 2m 56s [-1.25 s/s]; s3-rr: 3h 14m 58s [+1.00 s/s]; s3-user: 3h 14m 58s [+1.00 s/s]; s6-rr: 2h 57m 37s [+0.74 s/s]; s6-user: 2h 57m 37s [+0.74 s/s]; s7-rr: 9m 32s [+0.88 s/s]; s7-user: 9m 32s [+0.88 s/s] [23:52:21] 3(commented) [MNT-1183] Hyacinth crashed <10https://jira.toolserver.org/browse/MNT-1183> (DaB.) [23:52:47] @replag [23:52:52] DaBPunkt: s3-rr: 3h 28m 51s [+1.00 s/s]; s3-user: 3h 28m 55s [+1.00 s/s]; s6-rr: 3h 10m 21s [+0.90 s/s]; s6-user: 3h 10m 21s [+0.90 s/s]; s7-rr: 21m 9s [+0.83 s/s]; s7-user: 21m 10s [+0.83 s/s] [23:57:12] nacht ts