[00:10:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:19:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:41:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:49:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:01:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [01:06:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [01:09:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:11:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:14:34] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:19:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:31:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [01:42:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:49:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:19:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:37:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:45:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory [02:49:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:51:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [03:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:42:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:51:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:01:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [04:06:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [04:13:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:16:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [04:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:43:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:51:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:13:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:44:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:52:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:01:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [06:14:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:22:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:29:33] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:31:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:36:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [06:44:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:49:32] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:53:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:58:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 195 processes [07:11:02] PROBLEM SSH is now: CRITICAL on changefeed-bot i-0000041b.pmtpa.wmflabs output: CRITICAL - Socket timeout after 10 seconds [07:14:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:23:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:25:53] RECOVERY SSH is now: OK on changefeed-bot i-0000041b.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [07:44:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:53:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:14:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:23:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:44:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:54:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:14:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:24:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:37:02] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 10% free memory [09:44:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:54:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:01:37] Anyone know if there is a massblock for http://deployment.wikimedia.beta.wmflabs.org ? [10:01:42] !htmllogs [10:01:42] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [10:02:50] er [10:02:51] why? [10:03:04] is there spam? [10:03:22] Socks and spam yes [10:03:49] block them one by one? >.> [10:03:52] Do you know if there is a mass block script/tool? [10:03:54] write a script to do it [10:04:07] er [10:04:11] legoktm: There is like 50... [10:04:21] I am not smart enough to write a script. [10:04:23] if you give me a list of usernames, and +sysop i can do it ;) [10:04:40] provided api.php is enabled... [10:04:47] * Riley sighs, I'll do it. [10:04:59] I can't +sysop, only a gs for the cluster [10:05:21] can i poke someone to get it? [10:05:33] To get what? [10:05:45] +sysop [10:06:21] Probably. Try Timotheus_Canens [10:06:33] Don't think he is around at the moment. [10:06:54] Nono Thehelpfulone* [10:07:01] Pasted wrong name. [10:07:42] >.> [10:14:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:24:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:28:57] jeremyb: poke [10:44:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:54:32] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:14:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:24:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:45:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:54:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:15:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:20:32] PROBLEM Free ram is now: WARNING on swift-be4 i-000001ca.pmtpa.wmflabs output: Warning: 19% free memory [12:26:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:45:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:45:32] RECOVERY Free ram is now: OK on swift-be4 i-000001ca.pmtpa.wmflabs output: OK: 20% free memory [12:53:33] PROBLEM Free ram is now: WARNING on swift-be4 i-000001ca.pmtpa.wmflabs output: Warning: 19% free memory [12:56:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:15:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:22:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [13:26:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:45:42] PROBLEM dpkg-check is now: CRITICAL on maps-osmmapnik2 i-000004f2.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [13:46:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:50:42] RECOVERY dpkg-check is now: OK on maps-osmmapnik2 i-000004f2.pmtpa.wmflabs output: All packages OK [13:56:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:57:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 44% free memory [14:16:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:27:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:46:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:57:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:28:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:46:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:17:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:28:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:46:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [16:48:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:51:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [16:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:01:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [17:18:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:48:32] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 205 processes [17:49:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:53:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 196 processes [17:58:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:19:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:29:04] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:44:33] PROBLEM Current Users is now: CRITICAL on maps-osmmapnik i-00000527.pmtpa.wmflabs output: Connection refused by host [18:45:12] PROBLEM Disk Space is now: CRITICAL on maps-osmmapnik i-00000527.pmtpa.wmflabs output: Connection refused by host [18:45:52] PROBLEM Free ram is now: CRITICAL on maps-osmmapnik i-00000527.pmtpa.wmflabs output: Connection refused by host [18:45:52] PROBLEM Current Load is now: CRITICAL on maps-osmmapnik i-00000527.pmtpa.wmflabs output: Connection refused by host [18:47:22] PROBLEM Total processes is now: CRITICAL on maps-osmmapnik i-00000527.pmtpa.wmflabs output: Connection refused by host [18:48:12] PROBLEM dpkg-check is now: CRITICAL on maps-osmmapnik i-00000527.pmtpa.wmflabs output: Connection refused by host [18:49:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:59:32] RECOVERY Current Users is now: OK on maps-osmmapnik i-00000527.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:59:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:00:13] RECOVERY Disk Space is now: OK on maps-osmmapnik i-00000527.pmtpa.wmflabs output: DISK OK [19:00:53] RECOVERY Free ram is now: OK on maps-osmmapnik i-00000527.pmtpa.wmflabs output: OK: 3456% free memory [19:00:54] RECOVERY Current Load is now: OK on maps-osmmapnik i-00000527.pmtpa.wmflabs output: OK - load average: 0.20, 0.88, 0.84 [19:02:23] RECOVERY Total processes is now: OK on maps-osmmapnik i-00000527.pmtpa.wmflabs output: PROCS OK: 99 processes [19:03:13] RECOVERY dpkg-check is now: OK on maps-osmmapnik i-00000527.pmtpa.wmflabs output: All packages OK [19:19:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:22:02] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [19:29:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:36:22] PROBLEM dpkg-check is now: CRITICAL on maps-osmmapnik i-00000527.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [19:42:03] PROBLEM dpkg-check is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [19:50:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:56:52] RECOVERY dpkg-check is now: OK on kubo i-000003dd.pmtpa.wmflabs output: All packages OK [19:59:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:20:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:29:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:34:50] legoktm: yah? [20:50:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:01:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:20:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:32:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:50:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:02:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:21:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:32:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:46:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 10% free memory [22:47:32] jeremyb: last night riley h was complaining about how he had to block a bunch of spambots by hand on all the beta.wmflabs projects so i offered to help with my massblock scripts, but couldnt find a steward so he ended up doing them all by hand [22:48:24] ahh [22:52:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:02:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:33:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:53:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%