[00:14:54] Change on 12mediawiki a page Developer access was modified, changed by Jason.mcvetta link https://www.mediawiki.org/w/index.php?diff=611186 edit summary: [00:15:40] Ryan_Lane: so JS is compiled to the .NET bytecode? [00:15:50] * jeremyb wonders how rhino works [00:16:15] jeremyb: http://jurassic.codeplex.com/ [00:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:17:14] ok, so embedding JS in .NET [00:17:23] i guess that's not far off from rhino's function [00:19:36] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=611192 edit summary: /* User:Jason.mcvetta */ done [00:22:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:23:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:46:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:52:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:53:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:08:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:13:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:16:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:23:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:24:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:46:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [01:47:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:53:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:54:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:58:32] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [02:03:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [02:03:43] PROBLEM Free ram is now: CRITICAL on nova-precise1 i-00000236.pmtpa.wmflabs output: 3820868 [02:18:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:24:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:24:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:37:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:38:42] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3735856 [02:43:42] PROBLEM Free ram is now: CRITICAL on nova-precise1 i-00000236.pmtpa.wmflabs output: 3843672 [02:48:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:54:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:54:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:10:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory [03:18:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:18:32] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [03:24:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:24:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:38:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [03:48:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:54:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:55:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:17:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [04:18:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:24:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:26:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:28:42] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3092276 [04:48:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:55:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:56:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:19:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:26:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:26:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:29:43] PROBLEM Disk Space is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [05:30:12] PROBLEM SSH is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CRITICAL - Socket timeout after 10 seconds [05:30:22] PROBLEM Current Load is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [05:30:52] RECOVERY SSH is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [05:31:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 10% free memory [05:34:32] RECOVERY Disk Space is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: DISK OK [05:35:13] RECOVERY Current Load is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK - load average: 0.15, 1.15, 1.07 [05:49:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:56:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:57:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:08:52] PROBLEM Current Load is now: CRITICAL on dumpmuncher i-00000522.pmtpa.wmflabs output: Connection refused by host [06:09:32] PROBLEM Current Users is now: CRITICAL on dumpmuncher i-00000522.pmtpa.wmflabs output: Connection refused by host [06:10:13] PROBLEM Disk Space is now: CRITICAL on dumpmuncher i-00000522.pmtpa.wmflabs output: Connection refused by host [06:11:03] PROBLEM Free ram is now: CRITICAL on dumpmuncher i-00000522.pmtpa.wmflabs output: Connection refused by host [06:12:23] PROBLEM Total processes is now: CRITICAL on dumpmuncher i-00000522.pmtpa.wmflabs output: Connection refused by host [06:12:53] PROBLEM dpkg-check is now: CRITICAL on dumpmuncher i-00000522.pmtpa.wmflabs output: Connection refused by host [06:14:33] RECOVERY Current Users is now: OK on dumpmuncher i-00000522.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [06:15:13] RECOVERY Disk Space is now: OK on dumpmuncher i-00000522.pmtpa.wmflabs output: DISK OK [06:16:03] RECOVERY Free ram is now: OK on dumpmuncher i-00000522.pmtpa.wmflabs output: OK: 1193% free memory [06:17:23] RECOVERY Total processes is now: OK on dumpmuncher i-00000522.pmtpa.wmflabs output: PROCS OK: 83 processes [06:17:53] RECOVERY dpkg-check is now: OK on dumpmuncher i-00000522.pmtpa.wmflabs output: All packages OK [06:18:53] RECOVERY Current Load is now: OK on dumpmuncher i-00000522.pmtpa.wmflabs output: OK - load average: 0.03, 0.48, 0.49 [06:19:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:26:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:27:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:29:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:30:22] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 152 processes [06:31:32] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:39:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [06:41:01] Change on 12mediawiki a page Developer access was modified, changed by Ladquin link https://www.mediawiki.org/w/index.php?diff=611265 edit summary: [06:43:52] PROBLEM dpkg-check is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:46:32] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:48:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:48:52] RECOVERY dpkg-check is now: OK on kubo i-000003dd.pmtpa.wmflabs output: All packages OK [06:49:42] PROBLEM dpkg-check is now: CRITICAL on wikiminiatlas-2 i-0000044b.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:49:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:54:32] RECOVERY dpkg-check is now: OK on deployment-sql02 i-000004cb.pmtpa.wmflabs output: All packages OK [06:54:42] RECOVERY dpkg-check is now: OK on wikiminiatlas-2 i-0000044b.pmtpa.wmflabs output: All packages OK [06:57:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:00:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 145 processes [07:19:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:28:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:28:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:49:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:58:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:58:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:19:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:28:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:49:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:58:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:58:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:01:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [09:11:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 10.36, 9.03, 6.85 [09:16:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [09:20:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:28:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:28:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:31:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [09:39:23] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [09:39:33] PROBLEM Total processes is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [09:41:32] PROBLEM Current Users is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [09:41:32] PROBLEM SSH is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused [09:41:42] PROBLEM Free ram is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [09:41:42] PROBLEM Current Load is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [09:41:52] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [09:49:12] Hi! Anybody here= [09:49:13] ? [09:49:39] ^^ wikidata-dev-3 is broken [09:50:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:50:44] It wanted a reboot but did not come back afterwards. [09:50:58] Can anyone help? [09:51:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.62, 4.36, 4.86 [09:54:22] RECOVERY dpkg-check is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: All packages OK [09:54:32] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 117 processes [09:56:33] RECOVERY Current Users is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [09:56:33] RECOVERY SSH is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [09:56:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 98% free memory [09:56:43] RECOVERY Current Load is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK - load average: 0.21, 0.15, 0.07 [09:56:53] RECOVERY Disk Space is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: DISK OK [09:58:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:58:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:20:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:28:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:28:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:51:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:59:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:59:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:21:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:29:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:29:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:52:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:59:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:59:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:01:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [12:16:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [12:22:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:29:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:29:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:52:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:59:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:59:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:05:00] @seen hashar [13:05:00] petan: Last time I saw hashar they were quitting the network with reason: Quit: I am a manual virus, please copy me to your quit message. N/A at 11/28/2012 11:39:51 PM (1.13:25:08.4413290 ago) [13:06:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 4% free memory [13:23:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:29:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:29:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:59:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:59:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:20:03] Silke_WMDE and/or Silke_WMDE_ -- still around? [14:21:02] hi andrewbogott! yes. [14:21:56] I think I have some reviews assigned to you and you have some assigned to me :) [14:22:20] I haven't been working on puppet for 2 days -> other Wikidata excitements :) [14:22:48] Do you have time to give https://gerrit.wikimedia.org/r/#/c/35293/ a try? [14:22:51] But I saw the pending stuff [14:23:09] ah, I tried that one [14:23:36] did you try patchset 3? I tried to fix up the wikidata bits but it still isn't working for me quite [14:23:42] I had errors, but I had the impression they were not related to your changes [14:24:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:24:10] (Other timeout stuff that I couldn't follow-up since yesterday.) [14:24:18] Yeah... [14:24:39] will you be on IRC for a bit? I'll get set up and try to test it again right now. [14:24:54] Yes, I'll be here for a while. [14:25:12] OK, I'll login to my server and test it again, too [14:29:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:29:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:35:53] PROBLEM Current Load is now: CRITICAL on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: Connection refused by host [14:36:33] PROBLEM Current Users is now: CRITICAL on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: Connection refused by host [14:37:13] PROBLEM Disk Space is now: CRITICAL on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: Connection refused by host [14:37:23] PROBLEM Total processes is now: CRITICAL on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: Connection refused by host [14:37:53] PROBLEM dpkg-check is now: CRITICAL on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: Connection refused by host [14:38:03] PROBLEM Free ram is now: CRITICAL on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: Connection refused by host [14:45:52] RECOVERY Current Load is now: OK on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: OK - load average: 0.30, 0.77, 0.55 [14:46:32] RECOVERY Current Users is now: OK on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: USERS OK - 2 users currently logged in [14:47:12] RECOVERY Disk Space is now: OK on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: DISK OK [14:47:22] RECOVERY Total processes is now: OK on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: PROCS OK: 100 processes [14:48:02] RECOVERY Free ram is now: OK on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: OK: 644% free memory [14:52:53] RECOVERY dpkg-check is now: OK on mwreview-andrewdev3 i-00000523.pmtpa.wmflabs output: All packages OK [14:54:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:59:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:01:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:08:56] Silke_WMDE_: I'm getting an error with populateSitesTable.php -- does that happen to you too? [15:09:09] It's in the wikidata manifest but I'm unclear on if it's related to my change [15:09:28] I#ve got the same [15:10:08] * andrewbogott tries again minus the patch [15:10:31] * Silke_WMDE_ tries again with the patch (2nd ride) [15:19:05] Hm… seems to work fine on a second pass with or without the patch. [15:19:29] So I will leave it to you to determine if it's /really/ working or only appearing to work [15:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:27:22] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [15:29:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:31:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:33:19] andrewbogott: I think it's ok to give a +1 [15:34:23] andrewbogott: The other day I tried to rewrite the wikidata files with a defined type to install 2 mediawikis but that didn't work, yet. [15:37:29] Silke_WMDE_: OK, I'll merge those two patches. And, lemme study up on if/how to install multiple versions of a parameterized class [15:38:39] as far as I read, it's not possible at all. (http://docs.puppetlabs.com/learning/definedtypes.html) [15:42:32] PROBLEM Total processes is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [15:42:42] PROBLEM Free ram is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [15:44:32] PROBLEM Current Users is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [15:44:42] PROBLEM Current Load is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [15:44:52] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Connection refused by host [15:54:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:57:32] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 113 processes [15:57:42] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 224% free memory [15:59:32] RECOVERY Current Users is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [15:59:42] RECOVERY Current Load is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK - load average: 1.18, 0.86, 0.82 [15:59:52] RECOVERY Disk Space is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: DISK OK [15:59:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:01:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:05:23] Hm, so we'd need to change the singlenode class into a resource. [16:07:23] RECOVERY dpkg-check is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: All packages OK [16:08:17] * Silke_WMDE_ Aahhh! [16:09:45] Unicorn people, I need help. Our Wikidata dev server was still a lucid machine, I upgraded to precise and now I cannot reconnect via ssh (while in nagios everything is very green). [16:11:02] What do I do now? [16:14:12] Silke_WMDE_ hi [16:14:20] !console | Silke_WMDE_ [16:14:20] Silke_WMDE_: in case you want to see what is happening on terminal of your vm, check console output [16:14:25] hi [16:14:29] you really want to do that :) [16:14:42] I think that puppet is running or something [16:14:48] otherwise you can try rebooting it [16:15:19] @seerx Ryan_Lane [16:15:31] @seenrx Ryan_Lane [16:15:31] petan: Last time I saw Ryan_Lane they were quitting the network with reason: Quit: Leaving. at 11/30/2012 8:16:11 AM (07:59:20.1936980 ago) (multiple results were found: Ryan_Lane1) [16:21:11] petan: I keeps saying [16:21:11] * Starting puppet agent [80G [16:21:11] [74G[[31mfail[39;49m] [16:21:11] * Starting puppet agent [80G [16:21:11] [74G[[31mfail[39;49m] [16:21:33] ok, try rebooting it? [16:21:40] this is very likely the reason [16:21:47] i just did, same messages before and after rebooting [16:24:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:25:23] PROBLEM host: i-00000225.pmtpa.wmflabs is DOWN address: i-00000225.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:27:45] ^^ that's the one [16:29:43] RECOVERY host: i-00000225.pmtpa.wmflabs is UP address: i-00000225.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 9.83 ms [16:29:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:31:09] Apperently, it did not the LDAP server [16:31:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:31:51] +find [16:34:45] petan: So I would ask Ryan about this? [16:35:54] paravoid and andrewbogott might help you as well [16:36:10] I am pretty busy atm and I have idea how to fix this, other than reinstalling the box itself [16:36:13] * andrewbogott is clueless [16:36:15] * no idea [16:36:49] in fact, one could just ssh to machine, fix the puppet, run it, check the keys [16:36:57] but for that you would need to know the root password [16:37:02] and I don't even know if there is some [16:37:27] or, you could make Ryan mount the virtual disk to some other machine you can ssh to and fix it [16:37:43] but that is time consuming task :) dunno if he is willing to do that [16:37:51] I recommend you to e-mail him and !list to cc [16:37:52] i see [16:37:53] !list [16:37:53] we have a mailing list labs-l@lists.wikimedia.org feel free to send a message there, don't forget to subscribe [16:38:12] ok [16:38:36] thanks for the advice! [16:53:51] @infobot-detail list [16:53:51] !ping [16:53:51] pong [16:53:52] There is no such a key [16:54:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:56:54] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [16:59:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:02:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:24:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:26:13] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 154 processes [17:31:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:32:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:55:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:01:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:02:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:13:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [18:24:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.86, 8.68, 6.37 [18:25:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:31:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:32:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:38:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [18:43:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [18:45:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 6.29, 6.26, 5.35 [18:48:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.48, 6.54, 5.54 [18:55:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:58:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.85, 3.65, 4.52 [19:02:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:21:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.60, 5.85, 5.27 [19:25:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:31:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.78, 6.47, 5.56 [19:33:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:55:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:03:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:25:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:38:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [20:47:41] Change on 12mediawiki a page Developer access was modified, changed by William.gathoye link https://www.mediawiki.org/w/index.php?diff=611604 edit summary: [20:56:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:01:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.96, 4.44, 4.96 [21:03:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:11:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.95, 4.37, 4.94 [21:24:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.60, 5.66, 5.20 [21:26:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:27:14] Change on 12mediawiki a page Developer access was modified, changed by Milad Naseri link https://www.mediawiki.org/w/index.php?diff=611620 edit summary: [21:34:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:34:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.92, 4.41, 4.80 [21:36:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [21:44:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.02, 6.15, 5.48 [21:57:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:58:52] PROBLEM Current Load is now: CRITICAL on mobile-osm i-00000524.pmtpa.wmflabs output: Connection refused by host [21:59:32] PROBLEM Current Users is now: CRITICAL on mobile-osm i-00000524.pmtpa.wmflabs output: Connection refused by host [22:00:13] PROBLEM Disk Space is now: CRITICAL on mobile-osm i-00000524.pmtpa.wmflabs output: Connection refused by host [22:01:03] PROBLEM Free ram is now: CRITICAL on mobile-osm i-00000524.pmtpa.wmflabs output: Connection refused by host [22:02:23] PROBLEM Total processes is now: CRITICAL on mobile-osm i-00000524.pmtpa.wmflabs output: Connection refused by host [22:02:53] PROBLEM dpkg-check is now: CRITICAL on mobile-osm i-00000524.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [22:03:53] RECOVERY Current Load is now: OK on mobile-osm i-00000524.pmtpa.wmflabs output: OK - load average: 1.21, 1.19, 0.71 [22:04:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:04:33] RECOVERY Current Users is now: OK on mobile-osm i-00000524.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [22:05:13] RECOVERY Disk Space is now: OK on mobile-osm i-00000524.pmtpa.wmflabs output: DISK OK [22:06:03] RECOVERY Free ram is now: OK on mobile-osm i-00000524.pmtpa.wmflabs output: OK: 634% free memory [22:06:13] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 150 processes [22:07:24] RECOVERY Total processes is now: OK on mobile-osm i-00000524.pmtpa.wmflabs output: PROCS OK: 91 processes [22:07:53] RECOVERY dpkg-check is now: OK on mobile-osm i-00000524.pmtpa.wmflabs output: All packages OK [22:13:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [22:18:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [22:22:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.05, 4.70, 5.39 [22:27:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:29:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.26, 4.03, 4.69 [22:34:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:57:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:02:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.86, 4.97, 5.34 [23:04:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:15:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.15, 7.15, 6.64 [23:20:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 6.94, 6.39, 5.54 [23:28:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:34:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:50:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 5.04, 4.52, 4.98 [23:58:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%