[00:00:53] PROBLEM Current Load is now: CRITICAL on mobile-osm i-00000525.pmtpa.wmflabs output: Connection refused by host [00:00:53] PROBLEM Free ram is now: CRITICAL on mobile-osm i-00000525.pmtpa.wmflabs output: Connection refused by host [00:01:33] PROBLEM Current Users is now: CRITICAL on mobile-osm i-00000525.pmtpa.wmflabs output: Connection refused by host [00:02:13] PROBLEM Disk Space is now: CRITICAL on mobile-osm i-00000525.pmtpa.wmflabs output: Connection refused by host [00:02:23] PROBLEM Total processes is now: CRITICAL on mobile-osm i-00000525.pmtpa.wmflabs output: Connection refused by host [00:04:33] PROBLEM dpkg-check is now: CRITICAL on mobile-osm i-00000525.pmtpa.wmflabs output: Connection refused by host [00:04:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:08:52] PROBLEM Current Load is now: CRITICAL on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: Connection refused by host [00:09:32] PROBLEM Current Users is now: CRITICAL on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: Connection refused by host [00:09:32] RECOVERY dpkg-check is now: OK on mobile-osm i-00000525.pmtpa.wmflabs output: All packages OK [00:10:12] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: Connection refused by host [00:10:52] RECOVERY Current Load is now: OK on mobile-osm i-00000525.pmtpa.wmflabs output: OK - load average: 0.19, 0.76, 0.60 [00:10:52] RECOVERY Free ram is now: OK on mobile-osm i-00000525.pmtpa.wmflabs output: OK: 1198% free memory [00:11:32] RECOVERY Current Users is now: OK on mobile-osm i-00000525.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [00:11:52] PROBLEM Free ram is now: CRITICAL on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: Connection refused by host [00:12:12] RECOVERY Disk Space is now: OK on mobile-osm i-00000525.pmtpa.wmflabs output: DISK OK [00:12:22] PROBLEM Total processes is now: CRITICAL on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: Connection refused by host [00:12:22] RECOVERY Total processes is now: OK on mobile-osm i-00000525.pmtpa.wmflabs output: PROCS OK: 83 processes [00:12:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.73, 3.50, 4.59 [00:14:32] RECOVERY Current Users is now: OK on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [00:14:32] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [00:15:13] RECOVERY Disk Space is now: OK on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: DISK OK [00:16:53] RECOVERY Free ram is now: OK on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: OK: 2077% free memory [00:17:23] RECOVERY Total processes is now: OK on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: PROCS OK: 89 processes [00:18:53] RECOVERY Current Load is now: OK on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: OK - load average: 0.06, 0.49, 0.50 [00:19:33] RECOVERY dpkg-check is now: OK on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: All packages OK [00:28:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:34:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:35:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.49, 6.83, 5.65 [00:56:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [00:59:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:01:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [01:06:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:06:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:11:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:24:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 8.85, 7.43, 5.89 [01:29:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:36:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:56:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [01:59:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:06:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:29:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:37:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:40:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:54:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 19% free memory [02:54:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.45, 4.50, 5.00 [02:55:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.76, 4.06, 4.98 [02:59:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:06:34] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [03:07:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:15:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.73, 3.75, 4.56 [03:30:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:34:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [03:37:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:41:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:50:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.08, 4.39, 4.79 [04:00:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:07:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:12:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 17% free memory [04:30:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:37:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:08:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:31:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:38:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:56:53] anyone else having issues logging into labs? labsconsole keeps saying that I have cookies disabled ... [06:02:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:08:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:29:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:31:34] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:32:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:38:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:46:34] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:54:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [07:03:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:03:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [07:08:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:21:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [07:33:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:39:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:03:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:09:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:19:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.85, 4.20, 4.85 [08:33:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:39:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:03:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:09:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:34:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:39:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:04:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:11:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:34:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:41:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:57:33] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [11:02:32] RECOVERY dpkg-check is now: OK on wikidata-dev-7 i-00000526.pmtpa.wmflabs output: All packages OK [11:04:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:11:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:29:19] Change on 12mediawiki a page Developer access was modified, changed by Nick.townsend link https://www.mediawiki.org/w/index.php?diff=611857 edit summary: [11:34:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:41:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:04:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:06:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [12:11:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 7% free memory [12:12:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:34:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:42:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:56:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [13:05:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:13:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:36:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:43:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:06:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:11:24] Change on 12mediawiki a page Developer access was modified, changed by Satyaakam link https://www.mediawiki.org/w/index.php?diff=611875 edit summary: [14:11:54] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [14:12:32] Change on 12mediawiki a page Developer access was modified, changed by Satyaakam link https://www.mediawiki.org/w/index.php?diff=611876 edit summary: /* User:Satyaakam */ [14:13:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:36:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [14:37:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:44:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:54:37] would anybody like to have a look at this: https://labsconsole.wikimedia.org/wiki/Shell_Request/Bene ? [14:56:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [15:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:14:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:16:54] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [15:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:44:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:07:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:14:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:21:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [16:31:24] Change on 12mediawiki a page Developer access was modified, changed by Avocato link https://www.mediawiki.org/w/index.php?diff=611922 edit summary: [16:36:12] Change on 12mediawiki a page Developer access was modified, changed by Mwang link https://www.mediawiki.org/w/index.php?diff=611925 edit summary: /* Please create an account */ new section [16:38:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:46:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:08:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:10:47] Change on 12mediawiki a page Developer access was modified, changed by Edinwiki link https://www.mediawiki.org/w/index.php?diff=611932 edit summary: [17:15:05] !log metavidwiki metavidwiki installation notes: http://lists.wikimedia.org/pipermail/metavid-l/2012-December/000086.html [17:15:07] Logged the message, Master [17:16:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:38:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:47:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:17:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:38:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:47:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:09:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:10:13] Change on 12mediawiki a page Developer access was modified, changed by Milad Naseri link https://www.mediawiki.org/w/index.php?diff=611944 edit summary: /* Please create an account */ [19:18:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:26:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [19:39:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:48:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:00:28] 12/01/2012 - 20:00:28 - Creating a home directory for kentaur at /export/keys/kentaur [20:05:29] 12/01/2012 - 20:05:28 - Updating keys for kentaur at /export/keys/kentaur [20:09:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:18:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [20:33:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [20:39:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:49:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:09:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:19:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:39:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:49:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:09:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:19:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:39:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:49:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:09:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:19:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:40:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:49:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%