[00:00:27] <^demon|zzz> I found it: http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/adminlogbot/ [00:00:43] <^demon|zzz> I poked Roan to grab me the copy off linode, I believe it has some local hacks that never got committed. [00:00:53] <^demon|zzz> So, we'll convert, apply those hacks, and get it into git. [00:00:55] <^demon|zzz> :) [00:01:51] awesomesauce [00:01:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:24:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:30:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.39, 5.62, 5.15 [00:31:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:35:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.25, 4.05, 4.62 [00:36:21] ^demon|zzz, I just spent a long time tuning up the labs-logbot, no doubt working on an obsolete verison of the code. Let me know when it's in git and I'll try to reconcile my changes with git. [00:36:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 4.25, 5.02, 5.25 [00:43:10] <^demon|zzz> andrewbogott: The live copy on linode has wildly diverged from the copy in SVN. This will take some time to sort out. [00:44:15] * andrewbogott nods [00:54:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:01:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:05:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 17% free memory [01:06:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.54, 4.41, 4.84 [01:08:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:13:52] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:25:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:32:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:55:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:55:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 28% free memory [02:02:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:04:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.31, 6.36, 5.60 [02:19:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [02:25:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:26:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.94, 5.49, 5.15 [02:32:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:56:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:01:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.27, 3.86, 4.65 [03:03:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:04:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.06, 3.72, 4.87 [03:21:26] Change on 12mediawiki a page Developer access was modified, changed by Superm401 link https://www.mediawiki.org/w/index.php?diff=603208 edit summary: [03:26:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:33:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:56:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:04:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:26:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:34:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:56:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:59:06] Change on 12mediawiki a page Developer access was modified, changed by Sundar link https://www.mediawiki.org/w/index.php?diff=603269 edit summary: [05:04:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:27:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:34:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:52:51] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603272 edit summary: one created [05:57:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:00:41] Change on 12mediawiki a page Developer access was modified, changed by Jbk malkum link https://www.mediawiki.org/w/index.php?diff=603273 edit summary: [06:04:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:05:57] Change on 12mediawiki a page Developer access was modified, changed by Swaroopsm link https://www.mediawiki.org/w/index.php?diff=603274 edit summary: /* wikimedia bangalore */ new section [06:07:10] Change on 12mediawiki a page Developer access was modified, changed by Swaroopsm link https://www.mediawiki.org/w/index.php?diff=603275 edit summary: /* User:Swaroopsm */ [06:11:09] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603276 edit summary: /* User:Jbk malkum */ [06:11:26] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603277 edit summary: /* wikimedia bangalore */ [06:12:23] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603278 edit summary: /* User:Swaroopsm */ [06:12:41] Change on 12mediawiki a page Developer access was modified, changed by Swaroopsm link https://www.mediawiki.org/w/index.php?diff=603279 edit summary: /* User:Swaroopsm */ [06:14:08] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603281 edit summary: /* User:Jbk malkum */ [06:16:54] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603282 edit summary: /* User:Sundar */ [06:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:32:22] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:33:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [06:34:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:57:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 148 processes [06:58:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:04:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:28:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:34:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:58:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:05:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:09:02] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.85, 4.53, 4.97 [08:25:21] 11/10/2012 - 08:25:20 - Creating a home directory for mattflaschen at /export/keys/mattflaschen [08:28:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:30:22] 11/10/2012 - 08:30:22 - Updating keys for mattflaschen at /export/keys/mattflaschen [08:31:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.38, 4.69, 5.00 [08:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:06:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:59:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:06:01] Change on 12mediawiki a page Developer access was modified, changed by Rathnam.99 link https://www.mediawiki.org/w/index.php?diff=603341 edit summary: /* Git Account */ new section [10:06:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:08:52] PROBLEM Current Load is now: CRITICAL on mobile-varnish i-0000050a.pmtpa.wmflabs output: Connection refused by host [10:09:32] PROBLEM Current Users is now: CRITICAL on mobile-varnish i-0000050a.pmtpa.wmflabs output: Connection refused by host [10:10:12] PROBLEM Disk Space is now: CRITICAL on mobile-varnish i-0000050a.pmtpa.wmflabs output: Connection refused by host [10:10:52] PROBLEM Free ram is now: CRITICAL on mobile-varnish i-0000050a.pmtpa.wmflabs output: Connection refused by host [10:11:56] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603342 edit summary: /* User:Rathnam.99 */ created [10:12:22] PROBLEM Total processes is now: CRITICAL on mobile-varnish i-0000050a.pmtpa.wmflabs output: Connection refused by host [10:13:52] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [10:14:32] PROBLEM dpkg-check is now: CRITICAL on mobile-varnish i-0000050a.pmtpa.wmflabs output: Connection refused by host [10:18:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 99% free memory [10:20:12] RECOVERY Disk Space is now: OK on mobile-varnish i-0000050a.pmtpa.wmflabs output: DISK OK [10:20:52] RECOVERY Free ram is now: OK on mobile-varnish i-0000050a.pmtpa.wmflabs output: OK: 525% free memory [10:22:22] RECOVERY Total processes is now: OK on mobile-varnish i-0000050a.pmtpa.wmflabs output: PROCS OK: 84 processes [10:23:52] RECOVERY Current Load is now: OK on mobile-varnish i-0000050a.pmtpa.wmflabs output: OK - load average: 0.04, 0.66, 0.69 [10:24:32] RECOVERY Current Users is now: OK on mobile-varnish i-0000050a.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [10:24:32] RECOVERY dpkg-check is now: OK on mobile-varnish i-0000050a.pmtpa.wmflabs output: All packages OK [10:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:36:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:59:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:00:55] paravoid, we're having problems with account creation here in Bangalore: "There was either an authentication database error or you are not allowed to update your external account" [11:01:03] LDAP related? [11:06:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:07:54] andrewbogott_afk: you around? ^^ [11:15:39] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603352 edit summary: remove header [11:29:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:36:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:00:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:07:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:12:34] Change on 12mediawiki a page Developer access was modified, changed by Ershad link https://www.mediawiki.org/w/index.php?diff=603360 edit summary: [12:15:06] Change on 12mediawiki a page Developer access was modified, changed by Siebrand link https://www.mediawiki.org/w/index.php?diff=603362 edit summary: /* User:Ershad */ [12:30:03] Change on 12mediawiki a page Developer access was modified, changed by Ruud Koot link https://www.mediawiki.org/w/index.php?diff=603366 edit summary: [12:31:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:37:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:41:56] Change on 12mediawiki a page Developer access was modified, changed by Ramkrsna link https://www.mediawiki.org/w/index.php?diff=603367 edit summary: [12:44:19] Change on 12mediawiki a page Developer access was modified, changed by Ramkrsna link https://www.mediawiki.org/w/index.php?diff=603368 edit summary: /* User:Ramkrsna */ [13:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:05:34] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [13:07:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:10:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [13:17:37] Change on 12mediawiki a page Developer access was modified, changed by Krinkle link https://www.mediawiki.org/w/index.php?diff=603370 edit summary: /* User:Ruud Koot */ [13:21:10] Change on 12mediawiki a page Developer access was modified, changed by Reedy link https://www.mediawiki.org/w/index.php?diff=603371 edit summary: /* User:Ruud Koot */ done [13:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:37:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:40:52] PROBLEM Free ram is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host [13:40:52] PROBLEM Current Load is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host [13:41:32] PROBLEM Current Users is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host [13:42:22] PROBLEM Disk Space is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host [13:42:22] PROBLEM Total processes is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host [13:43:12] PROBLEM dpkg-check is now: CRITICAL on stackfarm-squid i-0000050c.pmtpa.wmflabs output: Connection refused by host [13:43:52] PROBLEM Total processes is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: PROCS WARNING: 151 processes [13:48:52] RECOVERY Total processes is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: PROCS OK: 148 processes [13:51:33] RECOVERY Current Users is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [13:52:13] RECOVERY Disk Space is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: DISK OK [13:52:23] RECOVERY Total processes is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: PROCS OK: 84 processes [13:53:14] RECOVERY dpkg-check is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: All packages OK [13:55:52] RECOVERY Free ram is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: OK: 1055% free memory [13:55:52] RECOVERY Current Load is now: OK on stackfarm-squid i-0000050c.pmtpa.wmflabs output: OK - load average: 0.16, 0.58, 0.62 [14:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:07:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:23:13] Grr, Error: invalid magic word 'ISONLINE' still occurs after long reported by many [14:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:38:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:02:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:29:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [15:32:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:38:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:02:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:08:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:39:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:03:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:09:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:33:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:39:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:03:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:09:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:33:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:34:57] For fucks sake [18:35:06] who rebuilt bots-apache1 to bots-apache01 [18:41:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:04:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [19:11:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:34:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:41:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:00:20] 11/10/2012 - 20:00:19 - Creating a home directory for ant at /export/keys/ant [20:02:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [20:05:22] 11/10/2012 - 20:05:22 - Updating keys for ant at /export/keys/ant [20:05:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [20:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:35:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:41:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:04:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [21:06:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:09:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 7.32, 6.57, 5.59 [21:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:17:03] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 11.22, 8.86, 6.57 [21:36:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:39:29] ping petan [21:39:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.91, 5.92, 5.33 [21:41:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:42:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.29, 6.28, 5.41 [21:45:51] !log deployment-prep Disabling OnlineStatusBar as a live hack. It was producing «Error: invalid magic word 'ISONLINE'» backtraces. Sorry, petan. [21:45:51] I can't open the cache file, which means I have pretty much no idea what projects exist. [21:46:05] an amnesic bot [21:52:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.63, 4.11, 4.74 [21:59:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.44, 4.73, 4.89 [22:02:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [22:06:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:11:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:12:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [22:27:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [22:27:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.88, 5.45, 5.11 [22:36:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:36:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.22, 5.48, 5.28 [22:42:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:06:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:12:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:37:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:42:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)