[00:13:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 19% free memory [00:18:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:19:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.64, 5.21, 5.07 [00:20:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:20:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.74, 5.41, 5.13 [00:24:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 5.00, 4.83, 4.90 [00:30:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.07, 4.03, 4.66 [00:48:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:51:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:52:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [01:09:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:14:33] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:18:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:21:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:47:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.02, 7.38, 5.68 [01:48:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.98, 6.93, 5.77 [01:49:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:51:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:53:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 27% free memory [01:56:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.80, 6.07, 5.36 [02:03:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.05, 3.91, 4.73 [02:13:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 6.51, 6.36, 6.12 [02:19:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:21:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:27:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.26, 5.31, 5.10 [02:31:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.68, 4.11, 4.86 [02:35:06] Hello, ping [02:36:02] On deployment.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin I get this error : "Deployment uses cookies to log in users. You have cookies disabled. Please enable them and try again." [02:36:22] so I cannot login on this wiki [02:40:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:49:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:51:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:53:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory [02:57:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.49, 3.70, 4.61 [03:03:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.92, 4.29, 4.85 [03:19:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:22:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:29:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:49:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:52:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:19:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:20:26] 11/22/2012 - 04:20:26 - Creating a home directory for eikes at /export/keys/eikes [04:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:25:34] 11/22/2012 - 04:25:34 - Updating keys for eikes at /export/keys/eikes [04:49:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:53:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:15:15] Change on 12mediawiki a page Developer access was modified, changed by Navdeep link https://www.mediawiki.org/w/index.php?diff=608325 edit summary: [05:17:09] Change on 12mediawiki a page Developer access was modified, changed by Navdeep link https://www.mediawiki.org/w/index.php?diff=608326 edit summary: /* User:Navdeep */ [05:19:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:23:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:49:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:09:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [06:19:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:21:12] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=608331 edit summary: /* User:Navdeep */ done [06:24:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:29:23] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:32:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:32:33] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:37:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [06:38:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:49:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:51:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:52:32] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [06:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:14:52] RECOVERY Free ram is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK: 34% free memory [07:14:52] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 80% free memory [07:14:52] RECOVERY Free ram is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK: 36% free memory [07:17:02] PROBLEM Total processes is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: PROCS CRITICAL: 314 processes [07:19:52] PROBLEM Total processes is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: PROCS CRITICAL: 497 processes [07:21:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:22:22] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: PROCS CRITICAL: 414 processes [07:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:43:00] Change on 12mediawiki a page Developer access was modified, changed by Ztugcesirin link https://www.mediawiki.org/w/index.php?diff=608340 edit summary: [07:51:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:54:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:57:13] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=608385 edit summary: /* User:Ztugcesirin */ done [08:11:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.24, 4.55, 4.86 [08:15:57] Change on 12mediawiki a page Developer access was modified, changed by Rtdwivedi link https://www.mediawiki.org/w/index.php?diff=608388 edit summary: [08:21:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:25:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:48:56] petan: when I do @RC+ for wm-bot, what is the code for commons? [08:49:17] @RC+ commons test [08:49:17] Inserted new item to feed of changes [08:49:23] @RC- commons test [08:49:23] Deleted item from feed [08:49:27] it's commons :D [08:49:42] heh thanks! [08:51:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:55:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:56:42] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: Critical: 5% free memory [09:04:03] PROBLEM Current Users is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [09:04:03] PROBLEM Total processes is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [09:06:22] PROBLEM Current Load is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [09:06:42] PROBLEM Disk Space is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [09:06:52] PROBLEM dpkg-check is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [09:08:52] RECOVERY Current Users is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [09:08:52] RECOVERY Total processes is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: PROCS OK: 101 processes [09:11:12] PROBLEM Current Load is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: WARNING - load average: 3.16, 7.25, 6.49 [09:11:32] RECOVERY Disk Space is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: DISK OK [09:11:42] RECOVERY dpkg-check is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: All packages OK [09:11:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 93% free memory [09:21:12] RECOVERY Current Load is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK - load average: 2.31, 3.21, 4.63 [09:21:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:26:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:51:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:56:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:14:35] Change on 12mediawiki a page Developer access was modified, changed by SHL link https://www.mediawiki.org/w/index.php?diff=608405 edit summary: [10:21:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:26:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:51:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:23:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:27:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:53:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:57:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:17:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [12:23:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:28:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:32:38] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=608442 edit summary: /* User:Rtdwivedi */ done [12:32:42] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=608443 edit summary: /* User:SHL */ deferred [12:35:35] there's a labs acct creation waiting that I can't do. (has SVN already) ^^^ [12:47:52] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 56% free memory [12:53:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:59:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:23:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:29:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:38:42] PROBLEM Free ram is now: WARNING on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: Warning: 16% free memory [13:43:43] RECOVERY Free ram is now: OK on wikidata-dev-2 i-00000259.pmtpa.wmflabs output: OK: 80% free memory [13:53:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:59:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:23:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:41:32] PROBLEM Current Users is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [14:46:32] PROBLEM Current Users is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:53:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:00:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:02:03] PROBLEM Current Load is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: WARNING - load average: 5.22, 5.29, 5.08 [15:24:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:30:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:40:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 18% free memory [15:54:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:00:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:24:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:31:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:36:26] Change on 12mediawiki a page Developer access was modified, changed by Orsagi link https://www.mediawiki.org/w/index.php?diff=608514 edit summary: [16:54:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:00:16] Association of Scientists, Developers and Faculties [17:00:24] was marked for deletion [17:00:41] can anyone guide me for relisting? [17:01:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:24:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:31:02] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [17:31:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:32:02] RECOVERY Current Load is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK - load average: 2.13, 4.40, 4.96 [17:37:53] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 17% free memory [17:37:54] PROBLEM Free ram is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Warning: 17% free memory [17:54:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:55:52] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 27% free memory [18:01:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:24:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:31:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:54:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:01:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:24:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:56:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:02:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:26:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:30:41] Anyone know how the Bots project's apache server is set up? It somehow serves from a directory shared by all the instances or something? [20:32:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:55:35] 11/22/2012 - 20:55:34 - Creating a home directory for odie5533 at /export/keys/odie5533 [20:55:48] Is having shell access and being a member of the Bastion project the same thing? [20:56:37] nevermind, that notice means I guess I have access [20:56:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:00:27] 11/22/2012 - 21:00:27 - Updating keys for odie5533 at /export/keys/odie5533 [21:03:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:03:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 12% free memory [21:08:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 40% free memory [21:15:24] If anyone's alive that can, I'd like to request access to the Bastion project. I am able to SSH to bastion, but from there it will not allow me to connect to instances. [21:18:37] can you log in there or not? [21:18:53] I can log in to Bastion, but I can not log in to an instance. [21:19:01] well, into the instance I created. [21:19:02] which instances? [21:19:07] webtools-odie [21:19:26] is it running? [21:19:34] when did you create it? [21:19:40] yesterday [21:19:51] so it should have had time to generate keys and everything [21:20:03] it should [21:20:39] I can login into webtools-odie [21:20:56] which is a bit odd since that's not my instance :P [21:21:00] I am receiving "Permission denied (publickey)" [21:22:39] are you in the webtools project? [21:22:44] Yes [21:23:42] I don't know what could be wrong [21:24:01] What did you type to access the instance? "ssh webtools-odie" ? [21:24:14] yeo [21:24:23] $ ssh webtools-odie -v [21:24:36] I can't sudo to take a look at /var/log/auth.log [21:24:43] which is odd again [21:25:00] maybe because you are the instancecreator_username [21:25:46] what does sudo::labs_project do? [21:26:09] from bastion? [21:26:25] do you mean sudo labs_project? [21:26:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:27:39] should I be uploading my SSH public key to my Bastion home directory? [21:28:20] wait, how are you connecting to bastion? [21:28:25] SSH [21:28:47] are you forwarding the ssh agent? [21:29:08] I enabled agent forwarding in Putty, yes. [21:30:54] and you have the ssh key loaded in puttyant ? [21:31:29] No [21:32:15] how were expecting bastion to connect with your ssh key, then? :) [21:32:32] I don't really know how bastion works xD [21:32:35] if ssh forwarding is enabled, but there's nothing to forward... [21:32:42] ah [21:33:05] another option is to create a new ssh key and store it on bastion [21:33:17] yay [21:33:31] Using pageant worked [21:33:41] Thank you [21:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:41:09] you're welcome [21:56:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:04:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:28:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:42:41] I accidentally created an instance without the proper security groups. I haven't done anything on the instance yet, so should I delete it and create a new one? [22:58:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:04:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:28:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:34:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:58:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)