[00:05:33] RECOVERY Disk Space is now: OK on wikistream-1 wikistream-1 output: DISK OK [00:20:43] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [00:50:43] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [01:20:43] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [01:50:43] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [02:20:43] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [02:20:54] turkey down! [02:37:53] PROBLEM Free ram is now: WARNING on deployment-web deployment-web output: Warning: 7% free memory [02:42:53] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: Critical: 5% free memory [02:50:43] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [03:20:43] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [03:50:43] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [03:55:57] PROBLEM Free ram is now: WARNING on orgcharts-dev orgcharts-dev output: Warning: 15% free memory [04:00:57] PROBLEM Free ram is now: WARNING on utils-abogott utils-abogott output: Warning: 14% free memory [04:00:57] PROBLEM Free ram is now: WARNING on nova-daas-1 nova-daas-1 output: Warning: 13% free memory [04:00:57] PROBLEM Free ram is now: WARNING on test-oneiric test-oneiric output: Warning: 14% free memory [04:03:27] PROBLEM Free ram is now: CRITICAL on test3 test3 output: Critical: 1% free memory [04:08:27] RECOVERY Free ram is now: OK on test3 test3 output: OK: 96% free memory [04:15:57] PROBLEM Free ram is now: CRITICAL on utils-abogott utils-abogott output: Critical: 5% free memory [04:15:57] PROBLEM Free ram is now: CRITICAL on orgcharts-dev orgcharts-dev output: Critical: 4% free memory [04:15:57] PROBLEM Free ram is now: CRITICAL on test-oneiric test-oneiric output: Critical: 5% free memory [04:20:57] PROBLEM Free ram is now: CRITICAL on nova-daas-1 nova-daas-1 output: Critical: 5% free memory [04:20:57] RECOVERY Free ram is now: OK on orgcharts-dev orgcharts-dev output: OK: 96% free memory [04:22:47] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [04:25:57] RECOVERY Free ram is now: OK on utils-abogott utils-abogott output: OK: 97% free memory [04:25:57] RECOVERY Free ram is now: OK on nova-daas-1 nova-daas-1 output: OK: 93% free memory [04:25:57] RECOVERY Free ram is now: OK on test-oneiric test-oneiric output: OK: 97% free memory [04:52:57] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [05:22:57] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [05:50:25] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 3.25, 4.49, 2.42 [05:51:43] PROBLEM Current Load is now: CRITICAL on deployment-web5 deployment-web5 output: CRITICAL - load average: 43.90, 30.15, 14.37 [05:53:23] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [05:55:23] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.31, 2.27, 2.05 [05:56:43] PROBLEM Current Load is now: WARNING on deployment-web5 deployment-web5 output: WARNING - load average: 2.12, 18.86, 14.73 [06:16:43] RECOVERY Current Load is now: OK on deployment-web5 deployment-web5 output: OK - load average: 0.30, 0.45, 4.10 [06:23:53] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [06:54:53] PROBLEM host: turnkey-1 is DOWN address: turnkey-1 CRITICAL - Host Unreachable (turnkey-1) [07:20:53] PROBLEM Puppet freshness is now: CRITICAL on aggregator1 aggregator1 output: Puppet has not run in the last 10 hours [07:20:53] PROBLEM Puppet freshness is now: CRITICAL on analytics analytics output: Puppet has not run in the last 10 hours [07:20:53] PROBLEM Puppet freshness is now: CRITICAL on analytics-main analytics-main output: Puppet has not run in the last 10 hours [07:20:53] PROBLEM Puppet freshness is now: CRITICAL on asher1 asher1 output: Puppet has not run in the last 10 hours [07:20:53] PROBLEM Puppet freshness is now: CRITICAL on backport backport output: Puppet has not run in the last 10 hours [21:36:00] petan: you about? [21:36:20] could you or someone else set up some more squids on deployment-prep? it's running so slowly.. [22:15:06] Thehelpfulone: I can't help you but ahm, can you explain why that would make it faster? [22:15:38] you're probably the only one or one of few people navigating wikis on beta.wmflabs, I don't think it's the stress on the squids slowing it down, is it? [22:15:48] well that's what hexmode told me last time? [22:15:54] ok [22:37:15] Thehelpfulone, what action is being slow? [22:37:37] loading pages in general Platonides [22:38:01] for example? [22:38:16] the web server load seems 0 [22:38:30] for example, http://labs.wikimedia.beta.wmflabs.org/wiki/Special:RecentChanges [22:39:31] how long does it take for you to load? I don't believe it's my connection as enwiki is working fine [22:41:07] not too much [22:41:09] for instance, -- [22:41:09] [23:44:25.301] GET http://labs.wikimedia.beta.wmflabs.org/wiki/Special:RecentChanges [HTTP/1.0 200 OK 2716ms] [22:41:52] hmm, now it seems to have hanged loading? [22:42:17] hmm, yeah it looks like I refreshed and it loaded quickly - but before it had been loading for probably about a minute.. [22:44:41] it seems to be a crawler there [22:45:31] wait, these access logs are those of yesterday [22:45:50] where are today logs? [22:48:02] I don't know, I don't have access to that project [22:49:34] the server is spawining runJobs on each request [22:49:38] that needs to be fixed [22:51:14] well, maybe it is coming from cron [23:15:02] Thehelpfulone, we're being crawled by google, the server is reaching MaxClients, and robots.txt is failing with a 500 [23:15:37] hmm, so multiple issues? [23:22:21] !log deployment-prep we're being crawled by google, the server is reaching MaxClients, and robots.txt is failing with a 500 [23:36:12] !log deployment-prep robots.txt fixed