[01:53:09] 2014/02/13 01:46 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2154 [01:53:09] 2014/02/13 01:46 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2140.000000 [02:19:10] 2014/02/13 02:18 OK z-dat-s5-b MySQL slave Uptime: 1700410 Threads: 4 Questions: 511434630 Slow queries: 1152 Opens: 801483 Flush tables: 2 Open tables: 239 Queries per second avg: 300.771 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1788 [02:20:10] 2014/02/13 02:19 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1595.000000 [02:22:10] 2014/02/13 02:21 CRIT hawthorn Sun Grid Engine execd CHECK_NRPE: Socket timeout after 30 seconds. [02:23:10] 2014/02/13 02:22 ?? hawthorn Sun Grid Engine execd option -w not recognized [05:36:06] [[Special:Log/newusers]] create 10 * Delphine664 * (New user account) [08:14:33] 2014/02/13 08:07 WARN nightshade Load avg. WARNING - load average: 19.55, 17.45, 13.35 [08:14:33] 2014/02/13 08:07 CRIT z-dat-s3-a MySQL slave Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [08:21:33] 2014/02/13 08:20 OK nightshade Load avg. OK - load average: 8.76, 14.37, 13.90 [08:39:35] 2014/02/13 08:32 WARN nightshade Load avg. WARNING - load average: 21.89, 18.99, 15.22 [08:52:35] 2014/02/13 08:51 CRIT nightshade Load avg. CRITICAL - load average: 25.15, 24.12, 20.33 [08:55:35] 2014/02/13 08:54 WARN nightshade Load avg. WARNING - load average: 12.15, 20.57, 19.77 [09:02:35] 2014/02/13 09:01 OK nightshade Load avg. OK - load average: 8.38, 10.17, 14.85 [09:04:35] 2014/02/13 08:57 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [09:17:36] 2014/02/13 09:10 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [09:22:37] 2014/02/13 09:15 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2064 [09:22:37] 2014/02/13 09:15 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2070.000000 [09:33:37] 2014/02/13 09:32 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.375 second response time [09:41:37] 2014/02/13 09:34 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [09:42:37] 2014/02/13 09:35 WARN yarrow Load avg. WARNING - load average: 23.16, 18.23, 10.84 [09:45:37] 2014/02/13 09:44 WARN wolfsbane toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.541 second response time [09:45:37] 2014/02/13 09:44 OK yarrow Load avg. OK - load average: 5.59, 14.38, 10.84 [09:46:37] 2014/02/13 09:45 CRIT wolfsbane toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.983 second response time [09:49:37] 2014/02/13 09:48 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.445 second response time [09:51:37] 2014/02/13 09:50 OK z-dat-s5-b MySQL slave Uptime: 1727530 Threads: 30 Questions: 518912084 Slow queries: 1357 Opens: 830008 Flush tables: 2 Open tables: 256 Queries per second avg: 300.378 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1596 [09:51:37] 2014/02/13 09:50 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1682.000000 [09:52:37] 2014/02/13 09:51 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [10:23:39] 2014/02/13 10:23 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3492 [10:34:40] 2014/02/13 10:34 OK z-dat-s3-a MySQL slave Uptime: 198309 Threads: 19 Questions: 63182892 Slow queries: 2769 Opens: 453780 Flush tables: 1 Open tables: 15963 Queries per second avg: 318.608 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1678 [11:07:22] [[Special:Log/newusers]] create 10 * Darlenemutter * (New user account) [11:09:16] [[User talk:Darlenemutter]] !N 10https://wiki.toolserver.org/w/index.php?oldid=8354&rcid=23015 * Darlenemutter * (+2796) (Created page with " The Thing That Makes Kingston the ideal Area to Resolve In? Whether you are planning or already living in Kingston rentals in Ontario,...") [13:39:45] 2014/02/13 13:39 OK cassia /sql DISK OK - free space: /sql 186252 MB (15% inode=99%): [13:40:45] 2014/02/13 13:39 WARN cassia /sql/data/dewiki DISK WARNING - free space: /sql 224965 MB (19% inode=99%): [13:41:45] 2014/02/13 13:40 OK cassia /sql/data/dewiki DISK OK - free space: /sql 353781 MB (29% inode=99%): [13:45:45] 2014/02/13 13:38 CRIT cassia s5 replag QUERY CRITICAL: Unknown database 'dewiki_p' [13:46:45] 2014/02/13 13:40 CRIT cassia MySQL slave Slave IO: No Slave SQL: No Seconds Behind Master: (null) [14:01:46] 2014/02/13 14:01 CRIT hawthorn Sun Grid Engine execd Connection refused by host [14:06:47] 2014/02/13 14:01 CRIT hawthorn / Connection refused by host [14:07:47] 2014/02/13 14:01 CRIT hawthorn /tmp Connection refused by host [14:07:47] 2014/02/13 14:01 CRIT hawthorn Environment IPMI Connection refused by host [14:07:47] 2014/02/13 14:00 CRIT hawthorn Load avg. Connection refused by host [14:55:50] 2014/02/13 14:54 CRIT wolfsbane / DISK CRITICAL - free space: / 3259 MB (10% inode=92%): [14:55:50] 2014/02/13 14:54 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3255 MB (10% inode=92%): [14:56:05] Marlen Caemmerer * [Toolserver-l] user-store and tile store failed again [15:03:50] 2014/02/13 15:03 OK hawthorn / DISK OK - free space: / 18477 MB (61% inode=93%): [15:03:51] 2014/02/13 15:03 OK hawthorn /tmp DISK OK - free space: /tmp 3636 MB (99% inode=97%): [15:03:51] 2014/02/13 15:03 OK hawthorn Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [15:03:51] 2014/02/13 15:03 ?? hawthorn Sun Grid Engine execd option -w not recognized [15:04:50] 2014/02/13 15:03 OK hawthorn Load avg. OK - load average: 0.60, 0.17, 0.07 [16:40:57] 2014/02/13 16:39 OK wolfsbane / DISK OK - free space: / 7026 MB (23% inode=93%): [16:40:57] 2014/02/13 16:39 OK wolfsbane /tmp DISK OK - free space: / 7023 MB (23% inode=93%): [18:56:40] omg, what happened to user store? [19:14:59] Danny_B: What do you mean? [19:28:08] 2014/02/13 19:24 WARN ortelius Load avg. WARNING - load average: 10.86, 19.35, 12.23 [19:28:08] 2014/02/13 19:22 CRIT wolfsbane Load avg. CRITICAL - load average: 14.46, 30.85, 18.83 [19:30:07] 2014/02/13 19:29 OK ortelius Load avg. OK - load average: 3.86, 14.47, 11.30 [19:32:12] scfc_de: i don't see any content there except for sk/ and wiwosm/ [19:39:07] 2014/02/13 19:37 CRIT ortelius Load avg. CRITICAL - load average: 98.27, 47.66, 26.26 [19:50:01] that's actually on willow. on nightshade i don't see any subdir [19:51:48] Danny_B: You didn't get http://permalink.gmane.org/gmane.org.wikimedia.toolserver/6528? [19:58:08] 2014/02/13 19:51 CRIT rosemary /mnt user-store DISK CRITICAL - free space: /mnt 17612 MB (0% inode=60%): [19:59:08] 2014/02/13 19:58 WARN wolfsbane Load avg. WARNING - load average: 8.83, 11.80, 19.68 [20:07:09] 2014/02/13 20:06 OK wolfsbane Load avg. OK - load average: 1.69, 7.67, 14.86 [20:10:09] 2014/02/13 20:09 WARN ortelius Load avg. WARNING - load average: 4.82, 7.46, 19.14 [20:11:09] 2014/02/13 20:10 WARN amaranth PING PING WARNING - Packet loss = 37%, RTA = 159.53 ms [20:13:09] 2014/02/13 20:13 OK amaranth PING PING OK - Packet loss = 0%, RTA = 156.55 ms [20:16:09] 2014/02/13 20:09 CRIT z-dat-s5-b MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:16:10] 2014/02/13 20:09 CRIT z-dat-s6-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:16:10] 2014/02/13 20:09 CRIT z-dat-s7-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:17:10] 2014/02/13 20:10 CRIT daphne MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:17:10] 2014/02/13 20:16 OK rosemary /mnt user-store DISK OK - free space: / 9973 MB (49% inode=91%): [20:17:10] 2014/02/13 20:10 CRIT rosemary MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:17:10] 2014/02/13 20:10 CRIT z-dat-s1-b MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:17:10] 2014/02/13 20:10 CRIT z-dat-s2-b MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:17:10] 2014/02/13 20:10 CRIT z-dat-s3-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:17:10] 2014/02/13 20:10 CRIT z-dat-s4-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [20:18:10] 2014/02/13 20:17 OK ortelius Load avg. OK - load average: 1.67, 6.59, 14.55 [20:22:09] 2014/02/13 20:21 CRIT amaranth PING PING CRITICAL - Packet loss = 70%, RTA = 216.54 ms [20:22:09] 2014/02/13 20:21 CRIT amaranth SMTP CRITICAL - Socket timeout after 10 seconds [20:23:09] 2014/02/13 20:22 OK amaranth SMTP SMTP OK - 1.995 sec. response time [20:23:09] 2014/02/13 20:22 CRIT amaranth SSH CRITICAL - Socket timeout after 10 seconds [20:25:09] 2014/02/13 20:24 CRIT wolfsbane / DISK CRITICAL - free space: / 3294 MB (10% inode=93%): [20:25:09] 2014/02/13 20:24 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3294 MB (10% inode=93%): [20:27:09] 2014/02/13 20:26 CRIT amaranth SSH CRITICAL - Socket timeout after 10 seconds [20:28:10] 2014/02/13 20:21 CRIT amaranth PING PING CRITICAL - Packet loss = 60%, RTA = 229.60 ms [20:28:11] 2014/02/13 20:27 CRIT amaranth SMTP CRITICAL - Socket timeout after 10 seconds [20:28:11] 2014/02/13 20:27 OK amaranth SSH SSH OK - OpenSSH_5.6p1-hpn13v10 (protocol 2.0) [20:34:11] 2014/02/13 20:34 OK amaranth PING PING OK - Packet loss = 16%, RTA = 154.95 ms [20:38:11] 2014/02/13 20:37 CRIT amaranth PING PING CRITICAL - Packet loss = 80%, RTA = 217.23 ms [20:39:11] 2014/02/13 20:37 WARN ortelius Load avg. WARNING - load average: 14.32, 19.20, 16.41 [20:41:12] 2014/02/13 20:34 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2171.000000 [20:41:12] 2014/02/13 20:34 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2188.000000 [20:41:12] 2014/02/13 20:34 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2201.000000 [20:42:12] 2014/02/13 20:41 OK ortelius Load avg. OK - load average: 3.71, 12.71, 14.36 [20:42:12] 2014/02/13 20:35 CRIT wolfsbane Load avg. CRITICAL - load average: 102.61, 68.06, 36.89 [20:43:12] 2014/02/13 20:41 CRIT web.amaranth PING PING CRITICAL - Packet loss = 80%, RTA = 155.04 ms [20:44:12] 2014/02/13 20:43 OK web.amaranth PING PING OK - Packet loss = 16%, RTA = 156.19 ms [20:46:12] 2014/02/13 20:45 CRIT amaranth SSH CRITICAL - Socket timeout after 10 seconds [20:47:13] 2014/02/13 20:46 OK amaranth SSH SSH OK - OpenSSH_5.6p1-hpn13v10 (protocol 2.0) [20:49:13] 2014/02/13 20:46 CRIT amaranth PING PING CRITICAL - Packet loss = 60%, RTA = 243.92 ms [20:50:13] 2014/02/13 20:49 WARN amaranth PING PING WARNING - Packet loss = 28%, RTA = 229.38 ms [20:52:13] 2014/02/13 20:51 CRIT amaranth PING PING CRITICAL - Packet loss = 60%, RTA = 222.66 ms [20:52:13] 2014/02/13 20:51 CRIT amaranth SSH CRITICAL - Socket timeout after 10 seconds [20:53:13] 2014/02/13 20:52 WARN amaranth PING PING WARNING - Packet loss = 44%, RTA = 131.40 ms [20:53:13] 2014/02/13 20:52 OK amaranth SSH SSH OK - OpenSSH_5.6p1-hpn13v10 (protocol 2.0) [20:54:13] 2014/02/13 20:53 CRIT amaranth PING PING CRITICAL - Packet loss = 70%, RTA = 224.97 ms [20:56:14] 2014/02/13 20:55 WARN wolfsbane Load avg. WARNING - load average: 1.07, 8.00, 19.70 [20:58:14] 2014/02/13 20:57 WARN amaranth PING PING WARNING - Packet loss = 44%, RTA = 225.87 ms [20:59:15] 2014/02/13 20:58 CRIT amaranth PING PING CRITICAL - Packet loss = 80%, RTA = 246.13 ms [21:01:15] 2014/02/13 21:01 OK amaranth PING PING OK - Packet loss = 0%, RTA = 123.44 ms [21:01:15] 2014/02/13 21:00 OK wolfsbane Load avg. OK - load average: 1.34, 3.77, 14.51 [21:05:15] 2014/02/13 21:04 CRIT daphne s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3611.000000 [21:05:15] 2014/02/13 21:04 CRIT rosemary s1 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3628.000000 [21:05:15] 2014/02/13 21:04 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3641.000000 [21:07:05] Marlen Caemmerer * Re: [Toolserver-l] user-store and tile store failed again [21:10:15] 2014/02/13 21:09 CRIT hemlock /tmp Connection refused by host [21:11:16] 2014/02/13 21:10 CRIT hemlock / Connection refused or timed out [21:11:16] 2014/02/13 21:10 CRIT hemlock /aux0 Connection refused or timed out [21:11:16] 2014/02/13 21:10 CRIT hemlock /home Connection refused or timed out [21:11:16] 2014/02/13 21:10 CRIT hemlock Load avg. Connection refused or timed out [21:11:16] 2014/02/13 21:10 CRIT hemlock NTP CRITICAL - Socket timeout after 10 seconds [21:11:16] 2014/02/13 21:10 CRIT hemlock PING CRITICAL - Host Unreachable (hemlock) [21:11:16] 2014/02/13 21:10 CRIT hemlock SMTP CRITICAL - Socket timeout after 10 seconds [21:11:17] 2014/02/13 21:10 CRIT hemlock SSH No route to host [21:12:17] 2014/02/13 21:11 CRIT hemlock Environment IPMI Connection refused or timed out [21:14:15] 2014/02/13 21:13 OK hemlock PING PING OK - Packet loss = 0%, RTA = 0.10 ms [21:16:15] 2014/02/13 21:15 WARN hemlock / DISK WARNING - free space: / 2997 MB (15% inode=87%): [21:16:15] 2014/02/13 21:15 WARN hemlock /tmp DISK WARNING - free space: / 2970 MB (14% inode=87%): [21:16:15] 2014/02/13 21:08 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [21:19:15] 2014/02/13 21:18 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3099 [21:19:16] 2014/02/13 21:18 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3425.000000 [21:20:15] 2014/02/13 21:19 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.010 second response time [21:21:16] 2014/02/13 21:20 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3527 [21:21:16] 2014/02/13 21:20 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3373 [21:21:16] 2014/02/13 21:20 OK z-dat-s5-b MySQL slave Uptime: 1768930 Threads: 16 Questions: 621520596 Slow queries: 1731 Opens: 832706 Flush tables: 2 Open tables: 256 Queries per second avg: 351.353 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 934 [21:21:16] 2014/02/13 21:20 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1097.000000 [21:21:16] 2014/02/13 21:20 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3534 [21:23:16] 2014/02/13 21:22 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3575 [21:23:16] 2014/02/13 21:22 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3579.000000 [21:23:16] 2014/02/13 21:22 WARN z-dat-s1-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3573 [21:23:16] 2014/02/13 21:22 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3273 [21:24:16] 2014/02/13 21:23 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3363 [21:25:16] 2014/02/13 21:24 OK z-dat-s3-a MySQL slave Uptime: 237309 Threads: 11 Questions: 78193438 Slow queries: 3100 Opens: 606967 Flush tables: 1 Open tables: 16384 Queries per second avg: 329.500 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1715 [21:25:16] 2014/02/13 21:24 OK z-dat-s6-a MySQL slave Uptime: 238567 Threads: 25 Questions: 59006468 Slow queries: 15791 Opens: 3634 Flush tables: 1 Open tables: 3362 Queries per second avg: 247.337 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1691 [21:26:16] 2014/02/13 21:25 OK z-dat-s2-b MySQL slave Uptime: 638683 Threads: 8 Questions: 171368564 Slow queries: 8525 Opens: 1961718 Flush tables: 1 Open tables: 256 Queries per second avg: 268.315 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1785 [21:26:16] 2014/02/13 21:25 OK z-dat-s4-a MySQL slave Uptime: 286837 Threads: 1 Questions: 35862517 Slow queries: 8 Opens: 91 Flush tables: 1 Open tables: 81 Queries per second avg: 125.27 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1696 [21:30:16] 2014/02/13 21:29 OK daphne MySQL slave Uptime: 561965 Threads: 10 Questions: 93088068 Slow queries: 7306 Opens: 6082 Flush tables: 1 Open tables: 1794 Queries per second avg: 165.647 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1701 [21:30:16] 2014/02/13 21:29 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1712.000000 [21:34:17] 2014/02/13 21:33 OK z-dat-s7-a MySQL slave Uptime: 237868 Threads: 23 Questions: 32672090 Slow queries: 11031 Opens: 5059 Flush tables: 1 Open tables: 4332 Queries per second avg: 137.353 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1740 [21:36:17] 2014/02/13 21:35 OK z-dat-s1-b MySQL slave Uptime: 612665 Threads: 3 Questions: 193044725 Slow queries: 2781 Opens: 1262 Flush tables: 1 Open tables: 208 Queries per second avg: 315.90 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1696 [21:40:17] 2014/02/13 21:39 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3569 [21:40:17] 2014/02/13 21:39 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3546.000000 [22:07:19] 2014/02/13 22:01 WARN hawthorn NTP NTP WARNING: Server has the LI_ALARM bit set, Offset 0.130716 secs [22:37:21] 2014/02/13 22:37 OK hawthorn NTP NTP OK: Offset 0.013007 secs [23:06:24] 2014/02/13 23:05 OK rosemary MySQL slave Uptime: 40353 Threads: 28 Questions: 36001698 Slow queries: 12435 Opens: 3923 Flush tables: 1 Open tables: 392 Queries per second avg: 892.169 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1794 [23:06:24] 2014/02/13 23:05 OK rosemary s1 replag QUERY OK: 'SELECT ts_rc_age()' returned 1764.000000