[00:57:03] 2013/07/17 00:50 CRIT ortelius Load avg. CRITICAL - load average: 92.46, 73.15, 36.57 [00:57:03] 2013/07/17 00:52 CRIT wolfsbane Load avg. CRITICAL - load average: 109.39, 92.15, 45.38 [01:19:05] 2013/07/17 01:18 WARN ortelius Load avg. WARNING - load average: 1.40, 5.27, 19.03 [01:22:04] 2013/07/17 01:21 WARN wolfsbane Load avg. WARNING - load average: 3.05, 5.05, 19.07 [01:24:05] 2013/07/17 01:23 OK ortelius Load avg. OK - load average: 1.84, 3.08, 14.18 [01:27:06] 2013/07/17 01:26 OK wolfsbane Load avg. OK - load average: 2.38, 3.46, 14.44 [04:30:13] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=8084&oldid=8067&rcid=22071 * 185.15.59.202 * (+0) (updated page) [06:28:26] 2013/07/17 06:27 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.063 second response time [06:29:27] 2013/07/17 06:28 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.250 second response time [07:23:30] 2013/07/17 07:19 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.634 second response time [07:24:31] 2013/07/17 07:23 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.873 second response time [07:25:31] 2013/07/17 07:24 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.477 second response time [08:08:32] 2013/07/17 08:07 WARN rosemary /mnt user-store DISK WARNING - free space: /mnt 355382 MB (6% inode=60%): [08:13:32] 2013/07/17 08:12 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.261 second response time [08:14:32] 2013/07/17 08:13 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.006 second response time [08:23:34] 2013/07/17 08:19 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.050 second response time [08:25:34] 2013/07/17 08:24 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.407 second response time [09:40:38] 2013/07/17 09:39 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3559.000000 [10:00:39] 2013/07/17 09:53 WARN ptolemy Load avg. WARNING - load average: 22.10, 18.05, 11.44 [10:04:40] 2013/07/17 10:03 OK ptolemy Load avg. OK - load average: 9.70, 14.34, 11.49 [10:19:41] 2013/07/17 10:12 CRIT nightshade aliasd Connection refused [10:28:42] 2013/07/17 10:27 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1777.000000 [11:04:45] 2013/07/17 11:02 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 9.791 second response time [11:07:44] 2013/07/17 11:06 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.008 second response time [11:21:45] 2013/07/17 11:18 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2022 [11:31:47] 2013/07/17 11:24 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2165.000000 [11:35:47] 2013/07/17 11:35 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.718 second response time [11:36:46] 2013/07/17 11:29 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2200 [11:36:47] 2013/07/17 11:35 CRIT z-dat-s7-a wikidata replag (Service Check Timed Out) [11:37:47] 2013/07/17 11:36 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2543.000000 [11:39:48] 2013/07/17 11:37 CRIT z-dat-s7-a wikidata replag (Service Check Timed Out) [11:40:48] 2013/07/17 11:32 CRIT z-dat-s6-a wikidata replag (Service Check Timed Out) [11:41:48] 2013/07/17 11:41 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.533 second response time [11:42:49] 2013/07/17 11:42 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.038 second response time [11:42:50] 2013/07/17 11:41 CRIT z-dat-s3-a MySQL slave (Service Check Timed Out) [11:42:50] 2013/07/17 11:41 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [11:42:50] 2013/07/17 11:41 CRIT z-dat-s7-a MySQL slave (Service Check Timed Out) [11:44:50] 2013/07/17 11:44 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.240 second response time [11:44:50] 2013/07/17 11:40 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [11:45:50] 2013/07/17 11:41 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [11:45:50] 2013/07/17 11:41 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [11:46:50] 2013/07/17 11:41 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [11:47:50] 2013/07/17 11:41 CRIT hyacinth / CHECK_NRPE: Socket timeout after 30 seconds. [11:47:50] 2013/07/17 11:41 CRIT hyacinth /tmp CHECK_NRPE: Socket timeout after 30 seconds. [11:47:50] 2013/07/17 11:41 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [11:47:50] 2013/07/17 11:41 CRIT hyacinth SSH CRITICAL - Socket timeout after 10 seconds [11:47:51] 2013/07/17 11:41 CRIT z-dat-s4-a Load avg. Timeout while attempting connection [11:47:51] 2013/07/17 11:41 CRIT z-dat-s6-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [11:47:51] 2013/07/17 11:41 CRIT z-dat-s7-a Load avg. CHECK_NRPE: Socket timeout after 30 seconds. [11:48:50] 2013/07/17 11:48 OK hyacinth / DISK OK - free space: / 6309 MB (21% inode=84%): [11:48:50] 2013/07/17 11:48 OK hyacinth /tmp DISK OK - free space: /tmp 530 MB (99% inode=99%): [11:48:50] 2013/07/17 11:48 OK hyacinth SMTP SMTP OK - 0.002 sec. response time [11:48:50] 2013/07/17 11:48 OK hyacinth SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:48:50] 2013/07/17 11:41 CRIT z-dat-s4-a /tmp Timeout while attempting connection [11:48:51] 2013/07/17 11:48 OK z-dat-s4-a Load avg. OK - load average: 0.39, 0.32, 0.79 [11:48:51] 2013/07/17 11:47 OK z-dat-s4-a MySQL Uptime: 172263 Threads: 1 Questions: 6518472 Slow queries: 2 Opens: 81 Flush tables: 1 Open tables: 71 Queries per second avg: 37.840 [11:48:51] 2013/07/17 11:47 OK z-dat-s4-a MySQL slave Uptime: 172263 Threads: 2 Questions: 6518468 Slow queries: 2 Opens: 81 Flush tables: 1 Open tables: 71 Queries per second avg: 37.840 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1305 [11:48:52] 2013/07/17 11:48 OK z-dat-s6-a /sql DISK OK - free space: /sql 293055 MB (36% inode=99%): [11:48:53] 2013/07/17 11:47 OK z-dat-s6-a MySQL Uptime: 442131 Threads: 72 Questions: 367295723 Slow queries: 25509 Opens: 93557 Flush tables: 1 Open tables: 3170 Queries per second avg: 830.739 [11:48:53] 2013/07/17 11:48 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2174 [11:48:53] 2013/07/17 11:48 OK z-dat-s7-a Load avg. OK - load average: 0.28, 0.30, 0.78 [11:49:50] 2013/07/17 11:48 OK z-dat-s3-a MySQL Uptime: 172857 Threads: 187 Questions: 335313084 Slow queries: 10826 Opens: 1667153 Flush tables: 1 Open tables: 16384 Queries per second avg: 1939.829 [11:49:50] 2013/07/17 11:48 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 65623 MB (10% inode=99%): [11:49:51] 2013/07/17 11:48 OK z-dat-s4-a /tmp DISK OK - free space: /tmp 531 MB (99% inode=99%): [11:49:51] 2013/07/17 11:48 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3235.000000 [11:50:50] 2013/07/17 11:49 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2227.000000 [11:56:51] 2013/07/17 11:55 CRIT z-dat-s7-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3653.000000 [12:00:52] 2013/07/17 11:59 CRIT z-dat-s7-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3621 [12:01:52] 2013/07/17 12:00 CRIT z-dat-s6-a wikidata replag (Service Check Timed Out) [12:02:52] 2013/07/17 12:01 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2956.000000 [12:09:45] [[Admin:Toolserver database]] 10https://wiki.toolserver.org/w/index.php?diff=8085&oldid=3364&rcid=22072 * Krinkle * (+94) (Add link to file in puppet) [12:09:53] 2013/07/17 12:06 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1901 [12:13:54] 2013/07/17 12:13 CRIT z-dat-s6-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3648 [12:13:54] 2013/07/17 12:12 CRIT z-dat-s6-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3626.000000 [12:25:54] 2013/07/17 12:22 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2023 [12:26:54] 2013/07/17 12:21 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.785 second response time [12:27:54] 2013/07/17 12:21 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2128.000000 [12:31:54] 2013/07/17 12:31 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.015 second response time [12:51:57] 2013/07/17 12:45 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2112.000000 [13:00:58] 2013/07/17 13:00 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3641 [13:00:58] 2013/07/17 13:00 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3657.000000 [13:10:58] 2013/07/17 13:10 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3567 [13:10:58] 2013/07/17 13:10 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1798.000000 [13:10:58] 2013/07/17 13:10 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3555.000000 [13:23:59] 2013/07/17 13:23 CRIT z-dat-s4-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3609 [13:32:59] 2013/07/17 13:32 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3598 [13:36:59] 2013/07/17 13:36 OK z-dat-s5-b MySQL slave Uptime: 436300 Threads: 6 Questions: 589838956 Slow queries: 3350 Opens: 388651 Flush tables: 1 Open tables: 256 Queries per second avg: 1351.911 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1687 [13:37:00] 2013/07/17 13:36 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1649.000000 [14:08:01] 2013/07/17 14:07 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3432 [14:11:02] 2013/07/17 14:10 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1746.000000 [14:12:01] 2013/07/17 14:11 OK z-dat-s5-b MySQL slave Uptime: 438400 Threads: 8 Questions: 590842896 Slow queries: 3368 Opens: 388730 Flush tables: 1 Open tables: 256 Queries per second avg: 1347.725 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1610 [14:13:02] 2013/07/17 14:12 OK z-dat-s4-a MySQL slave Uptime: 180902 Threads: 3 Questions: 6932360 Slow queries: 3 Opens: 81 Flush tables: 1 Open tables: 71 Queries per second avg: 38.321 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1788 [14:41:03] 2013/07/17 14:40 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3595 [14:42:03] 2013/07/17 14:41 CRIT z-dat-s6-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3624 [14:43:03] 2013/07/17 14:42 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3496 [14:50:04] 2013/07/17 14:49 OK z-dat-s6-a MySQL slave Uptime: 453019 Threads: 9 Questions: 372198284 Slow queries: 33942 Opens: 101212 Flush tables: 1 Open tables: 3227 Queries per second avg: 821.595 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1753 [15:00:03] 2013/07/17 14:52 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2111.000000 [15:05:04] 2013/07/17 15:03 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3593 [15:12:05] 2013/07/17 15:05 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2190 [15:13:04] 2013/07/17 15:11 OK z-dat-s7-a MySQL slave Uptime: 185067 Threads: 15 Questions: 168463476 Slow queries: 12425 Opens: 375750 Flush tables: 1 Open tables: 3555 Queries per second avg: 910.283 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1709 [15:27:05] 2013/07/17 15:25 CRIT z-dat-s2-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3656.000000 [15:34:06] 2013/07/17 15:27 WARN daphne wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2021.000000 [15:36:08] 2013/07/17 15:35 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [16:08:10] 2013/07/17 16:07 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3517 [16:13:04] DaB. * Re: [Toolserver-announce] Short downtime of toolserver.org at Wednesday [16:20:09] 2013/07/17 16:19 OK z-dat-s3-a MySQL slave Uptime: 189126 Threads: 27 Questions: 347110124 Slow queries: 43039 Opens: 1693181 Flush tables: 1 Open tables: 16383 Queries per second avg: 1835.337 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1684 [16:48:14] 2013/07/17 16:47 CRIT daphne wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3626.000000 [17:47:18] 2013/07/17 17:46 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3574.000000 [18:10:18] 2013/07/17 18:08 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.552 second response time [18:11:18] 2013/07/17 18:10 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.661 second response time [18:15:17] 2013/07/17 18:14 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.083 second response time [18:22:17] 2013/07/17 18:21 OK z-dat-s6-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1783.000000 [18:35:17] 2013/07/17 18:34 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3417 [18:39:18] 2013/07/17 18:38 OK z-dat-s2-b MySQL slave Uptime: 5134535 Threads: 12 Questions: 4880986933 Slow queries: 89072 Opens: 38077567 Flush tables: 1 Open tables: 256 Queries per second avg: 950.619 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1786 [19:23:21] 2013/07/17 19:22 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3595.000000 [19:59:24] 2013/07/17 19:52 WARN ortelius / DISK WARNING - free space: / 5530 MB (18% inode=92%): [20:03:24] 2013/07/17 20:02 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1791.000000 [21:10:28] 2013/07/17 21:08 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.645 second response time [21:11:28] 2013/07/17 21:10 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.208 second response time [21:24:29] 2013/07/17 21:23 CRIT ortelius / DISK CRITICAL - free space: / 3131 MB (10% inode=92%): [21:24:29] 2013/07/17 21:23 CRIT wolfsbane / DISK CRITICAL - free space: / 3285 MB (10% inode=93%): [21:24:29] 2013/07/17 21:24 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3289 MB (10% inode=93%): [21:25:29] 2013/07/17 21:24 WARN wolfsbane / DISK WARNING - free space: / 3300 MB (11% inode=93%): [21:25:29] 2013/07/17 21:25 WARN wolfsbane /tmp DISK WARNING - free space: / 3299 MB (11% inode=93%): [21:26:29] 2013/07/17 21:25 CRIT wolfsbane / DISK CRITICAL - free space: / 3284 MB (10% inode=93%): [21:26:29] 2013/07/17 21:26 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3271 MB (10% inode=93%): [21:30:29] 2013/07/17 21:30 OK wolfsbane /tmp DISK OK - free space: / 6605 MB (22% inode=93%): [21:31:30] 2013/07/17 21:30 OK wolfsbane / DISK OK - free space: / 6596 MB (22% inode=93%): [22:17:32] 2013/07/17 22:10 WARN wolfsbane / DISK WARNING - free space: / 6105 MB (20% inode=93%): [22:17:32] 2013/07/17 22:11 WARN wolfsbane /tmp DISK WARNING - free space: / 6060 MB (20% inode=93%):