[00:07:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:24:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:54:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:08:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 189 processes [01:13:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:25:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:27:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [01:39:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:56:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:09:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:26:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:39:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:40:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:53:10] Change on 12mediawiki a page Developer access was modified, changed by Aracele link https://www.mediawiki.org/w/index.php?diff=606387 edit summary: [02:54:53] Change on 12mediawiki a page Developer access was modified, changed by Aracele link https://www.mediawiki.org/w/index.php?diff=606388 edit summary: /* User:Aracele */ [02:56:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:58:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory [03:09:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:26:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:34:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.80, 5.48, 5.20 [03:39:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs CRITICAL - Plugin timed out after 10 seconds [03:43:32] RECOVERY Free ram is now: OK on swift-be2 i-000001c8.pmtpa.wmflabs output: OK: 21% free memory [03:49:09] petan, Damianz: would it be possible to install curl on bots.wmflabs.org? [03:54:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.40, 4.55, 4.87 [03:56:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:10:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:26:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:40:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:49:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 3.97, 5.21, 5.16 [04:56:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:10:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:26:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:41:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:56:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:01:32] PROBLEM Free ram is now: WARNING on swift-be2 i-000001c8.pmtpa.wmflabs output: Warning: 19% free memory [06:11:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:14:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [06:19:44] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [06:26:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:28:34] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:31:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:32:23] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:32:33] RECOVERY dpkg-check is now: OK on integration-apache1 i-000002eb.pmtpa.wmflabs output: All packages OK [06:35:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 152 processes [06:36:02] PROBLEM Free ram is now: CRITICAL on mobile-solr2 i-00000507.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [06:40:32] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 146 processes [06:40:52] RECOVERY Free ram is now: OK on mobile-solr2 i-00000507.pmtpa.wmflabs output: OK: 92% free memory [06:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:42:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:46:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [06:52:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.67, 4.77, 5.00 [06:53:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:56:33] RECOVERY Free ram is now: OK on swift-be2 i-000001c8.pmtpa.wmflabs output: OK: 21% free memory [06:58:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:03:43] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [07:04:33] PROBLEM Free ram is now: WARNING on swift-be2 i-000001c8.pmtpa.wmflabs output: Warning: 19% free memory [07:10:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.56, 5.48, 5.17 [07:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:28:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:41:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:58:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:28:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:41:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:58:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:11:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:20:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.05, 4.33, 4.77 [09:24:32] RECOVERY Free ram is now: OK on swift-be2 i-000001c8.pmtpa.wmflabs output: OK: 21% free memory [09:28:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:37:32] PROBLEM Free ram is now: WARNING on swift-be2 i-000001c8.pmtpa.wmflabs output: Warning: 19% free memory [09:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:46:33] RECOVERY Free ram is now: OK on swift-be4 i-000001ca.pmtpa.wmflabs output: OK: 34% free memory [09:58:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:28:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:42:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:52:32] RECOVERY Free ram is now: OK on swift-be2 i-000001c8.pmtpa.wmflabs output: OK: 38% free memory [10:59:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:13:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:29:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:43:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:59:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:13:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:29:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:43:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:46:19] Huh, having some problems logging into labswiki. It complains that I have cookies disabled, yet it's able to set the session cookie just fine. [12:59:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:14:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:31:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:44:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:01:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:14:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:31:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:44:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:01:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:14:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs CRITICAL - Plugin timed out after 10 seconds [15:31:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:44:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:01:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:14:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:30:15] Damianz: maybe you can help me with this. Just tried to create a Labs account and got this error: "The user account was not created, as we could not confirm its source. Ensure you have cookies enabled, reload this page and try again. " [16:31:14] That sounds like andrewbogott_afk needs to kick the memcache server again as it's not storing your session cookie value so mediawiki thinks you don't have cookies when you do. [16:31:45] lemme try logging out and in again [16:32:18] (labsconsole response is slower than I am used to) [16:33:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:33:43] also the first time I try to log in to it or to mediawiki.org I get the "you have cookies disabled" error and then I try again and it works on mediawiki.org but not on labsconsole [16:34:03] Damianz: could you do me the favor of making aracele's account (bottom of https://www.mediawiki.org/wiki/Developer_access ) while we get this sorted? [16:35:30] I don't think I have access to and I don't have my phone with me to login atm, sorry [16:43:09] ok, thanks Damianz [16:44:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:47:46] paravoid: andrewbogott_afk - could one of you help with this problem? I have cookies turned on, but I cannot log into labsconsole because it thinks I have cookies turned off [17:03:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:14:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:15:08] drdee: hi! how are you? how is your family? [17:15:22] hey, we are doing awesome! [17:33:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:45:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:03:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:15:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:33:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:43:32] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [18:46:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:48:34] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [19:03:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:16:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:34:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:46:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:04:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:16:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:34:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:47:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:03:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 152 processes [21:04:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:08:33] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 148 processes [21:17:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:34:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:47:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:06:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:18:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:18:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [22:23:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [22:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:48:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:06:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:19:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:36:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:49:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%