[00:05:21] yeah [00:05:29] not totally sure if we'll use that yet [00:05:46] it would perform better [00:05:53] but it's not safe to give out root in containers [00:09:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:09:51] gwicke: heh. you can definitely see your rt run: http://ganglia.wikimedia.org/latest/?c=Virtualization%20cluster%20pmtpa&h=virt8.pmtpa.wmnet&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [00:11:04] I'm also force running puppet on all nodes, but I'm running that in batches of 5, though [00:11:32] <3 salt [00:12:40] I need to add the rest of the virt hosts into the cluster... [00:16:05] 10/26/2012 - 00:16:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:16:52] PROBLEM Free ram is now: CRITICAL on changefeed-bot i-0000041b.pmtpa.wmflabs output: Critical: 5% free memory [00:20:14] Ryan_Lane: definitely eating some cpu time there ;) [00:20:25] :) [00:20:27] we still have some breathing room ;) [00:20:34] and we have 3 more nodes to add [00:20:57] we'll consume them if we get our hands on them ;) [00:21:01] 10/26/2012 - 00:21:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:21:02] :D [00:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:26:06] 10/26/2012 - 00:26:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:27:29] [bz] (8RESOLVED - created by: 2Ryan Lane, priority: 4Unprioritized - 6normal) [Bug 40028] Remove the m1.tiny flavor - https://bugzilla.wikimedia.org/show_bug.cgi?id=40028 [00:27:45] whoeever made the bug bot write in here has my thanks :) [00:28:22] RECOVERY Disk Space is now: OK on maps-test-puppetosm2 i-000004d9.pmtpa.wmflabs output: DISK OK [00:31:02] 10/26/2012 - 00:30:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:36:02] 10/26/2012 - 00:36:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:39:44] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:41:01] 10/26/2012 - 00:41:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:41:19] 10/26/2012 - 00:41:18 - Creating a home directory for bean49 at /export/keys/bean49 [00:43:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [00:46:00] 10/26/2012 - 00:45:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:46:17] 10/26/2012 - 00:46:15 - Updating keys for bean49 at /export/keys/bean49 [00:48:54] PROBLEM Current Load is now: CRITICAL on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: Connection refused by host [00:49:34] PROBLEM Current Users is now: CRITICAL on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: Connection refused by host [00:50:14] PROBLEM Disk Space is now: CRITICAL on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: Connection refused by host [00:50:54] PROBLEM Free ram is now: CRITICAL on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: Connection refused by host [00:51:00] 10/26/2012 - 00:51:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:51:24] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 160 processes [00:52:24] PROBLEM Total processes is now: CRITICAL on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: Connection refused by host [00:54:34] PROBLEM dpkg-check is now: CRITICAL on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: Connection refused by host [00:54:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:55:13] RECOVERY Disk Space is now: OK on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: DISK OK [00:55:57] RECOVERY Free ram is now: OK on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: OK: 1115% free memory [00:56:07] 10/26/2012 - 00:56:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [00:57:23] RECOVERY Total processes is now: OK on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: PROCS OK: 84 processes [00:58:53] RECOVERY Current Load is now: OK on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: OK - load average: 0.12, 0.59, 0.60 [00:59:33] RECOVERY Current Users is now: OK on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [00:59:33] RECOVERY dpkg-check is now: OK on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: All packages OK [01:01:04] 10/26/2012 - 01:01:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:06:08] 10/26/2012 - 01:06:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:09:02] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 172 processes [01:10:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:11:02] 10/26/2012 - 01:11:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:14:03] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 95 processes [01:16:08] 10/26/2012 - 01:16:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:21:05] 10/26/2012 - 01:21:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:22:33] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [01:25:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:26:03] 10/26/2012 - 01:26:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:31:07] 10/26/2012 - 01:31:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:36:03] 10/26/2012 - 01:36:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:41:01] 10/26/2012 - 01:41:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:41:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:42:13] PROBLEM dpkg-check is now: CRITICAL on ve-change-marking i-000004e0.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [01:42:57] PROBLEM Current Load is now: CRITICAL on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Connection refused by host [01:46:02] 10/26/2012 - 01:46:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:46:14] PROBLEM Disk Space is now: CRITICAL on sube i-000003d0.pmtpa.wmflabs output: DISK CRITICAL - free space: / 39 MB (2% inode=40%): [01:47:15] RECOVERY dpkg-check is now: OK on ve-change-marking i-000004e0.pmtpa.wmflabs output: All packages OK [01:47:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.95, 7.43, 7.34 [01:51:06] 10/26/2012 - 01:51:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [01:55:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:56:03] 10/26/2012 - 01:56:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:01:16] 10/26/2012 - 02:01:15 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:06:12] 10/26/2012 - 02:06:12 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:11:07] 10/26/2012 - 02:11:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:11:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:16:03] 10/26/2012 - 02:16:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:19:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [02:21:00] 10/26/2012 - 02:21:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:26:01] 10/26/2012 - 02:26:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:26:02] RECOVERY host: i-000004de.pmtpa.wmflabs is UP address: i-000004de.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.63 ms [02:27:52] RECOVERY Disk Space is now: OK on patchtest i-000000f1.pmtpa.wmflabs output: DISK OK [02:27:52] RECOVERY Current Load is now: OK on patchtest i-000000f1.pmtpa.wmflabs output: OK - load average: 0.00, 0.00, 0.02 [02:28:22] RECOVERY Total processes is now: OK on patchtest i-000000f1.pmtpa.wmflabs output: PROCS OK: 83 processes [02:28:32] RECOVERY dpkg-check is now: OK on patchtest2 i-000000fd.pmtpa.wmflabs output: All packages OK [02:28:42] RECOVERY Current Users is now: OK on patchtest i-000000f1.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [02:28:42] RECOVERY Free ram is now: OK on patchtest i-000000f1.pmtpa.wmflabs output: OK: 628% free memory [02:31:05] 10/26/2012 - 02:31:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:32:51] Bahhhh. [02:33:17] * Hello71 grumbles [02:33:34] All I want to do is add a localization for an extension without going through translatewiki.net... [02:33:58] Never mind. [02:34:11] Note to self: To avoid looking stupid, read all messages printed by remote. [02:34:49] New question. [02:34:59] If I did something stupid but pushed already, what do I do? [02:35:12] Amend the commit but keep the Change-Id? [02:35:42] !htmllogs [02:35:42] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [02:35:52] PROBLEM Current Load is now: CRITICAL on patchtest i-000000f1.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [02:36:04] 10/26/2012 - 02:36:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:36:04] PROBLEM Disk Space is now: CRITICAL on patchtest i-000000f1.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [02:36:34] PROBLEM Total processes is now: CRITICAL on patchtest i-000000f1.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [02:36:42] PROBLEM dpkg-check is now: CRITICAL on patchtest2 i-000000fd.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [02:36:52] PROBLEM Current Users is now: CRITICAL on patchtest i-000000f1.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [02:36:52] PROBLEM Free ram is now: CRITICAL on patchtest i-000000f1.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [02:37:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:38:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:39:43] Ookay, looks like the workflow is amend, then git-review again. [02:39:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [02:41:21] 10/26/2012 - 02:41:20 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:41:57] RECOVERY Free ram is now: OK on changefeed-bot i-0000041b.pmtpa.wmflabs output: OK: 34% free memory [02:42:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:42:27] Nobody ever says anything... [02:42:27] :( [02:45:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 19% free memory [02:46:01] 10/26/2012 - 02:46:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:49:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [02:51:03] 10/26/2012 - 02:51:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:56:01] 10/26/2012 - 02:56:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [02:58:26] Hm... anyone working on labs atm? [02:58:43] s/labs/beta [03:01:04] 10/26/2012 - 03:01:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:06:00] 10/26/2012 - 03:05:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:09:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:11:06] 10/26/2012 - 03:11:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:12:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:16:02] 10/26/2012 - 03:16:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:21:05] 10/26/2012 - 03:21:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:26:04] 10/26/2012 - 03:26:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:31:05] 10/26/2012 - 03:31:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:36:08] 10/26/2012 - 03:36:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:39:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:41:00] 10/26/2012 - 03:41:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:42:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:46:03] 10/26/2012 - 03:46:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:51:09] 10/26/2012 - 03:51:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [03:54:43] PROBLEM dpkg-check is now: CRITICAL on maps-test-puppetosm2 i-000004d9.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [03:56:05] 10/26/2012 - 03:56:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:01:01] 10/26/2012 - 04:00:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:03:04] Change on 12mediawiki a page Developer access was modified, changed by Nandakv4 link https://www.mediawiki.org/w/index.php?diff=597639 edit summary: [04:06:00] 10/26/2012 - 04:05:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:10:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:11:06] 10/26/2012 - 04:11:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:13:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:16:05] 10/26/2012 - 04:16:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:19:37] RECOVERY dpkg-check is now: OK on maps-test-puppetosm2 i-000004d9.pmtpa.wmflabs output: All packages OK [04:21:06] 10/26/2012 - 04:21:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:26:01] 10/26/2012 - 04:26:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:31:05] 10/26/2012 - 04:31:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:32:54] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.57, 4.75, 4.96 [04:36:01] 10/26/2012 - 04:36:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:40:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:41:01] 10/26/2012 - 04:41:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:43:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:46:02] 10/26/2012 - 04:45:58 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:51:09] 10/26/2012 - 04:51:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [04:55:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.20, 5.26, 5.11 [04:56:02] 10/26/2012 - 04:56:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:01:15] 10/26/2012 - 05:01:15 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:06:09] 10/26/2012 - 05:06:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:10:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:11:09] 10/26/2012 - 05:11:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:13:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:16:02] 10/26/2012 - 05:16:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:21:02] 10/26/2012 - 05:21:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:22:33] PROBLEM dpkg-check is now: CRITICAL on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [05:26:05] 10/26/2012 - 05:26:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:31:01] 10/26/2012 - 05:31:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:32:35] RECOVERY dpkg-check is now: OK on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: All packages OK [05:36:12] 10/26/2012 - 05:36:12 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:40:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:41:00] 10/26/2012 - 05:41:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:43:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:46:05] 10/26/2012 - 05:46:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:50:54] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.77, 4.81, 4.99 [05:51:04] 10/26/2012 - 05:51:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [05:51:34] PROBLEM dpkg-check is now: CRITICAL on maps-test-puppetosm3 i-000004ee.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [05:56:05] 10/26/2012 - 05:56:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:01:02] 10/26/2012 - 06:01:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:06:05] 10/26/2012 - 06:06:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:11:04] 10/26/2012 - 06:11:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:11:17] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:14:16] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:16:09] 10/26/2012 - 06:16:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:21:02] 10/26/2012 - 06:21:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:26:01] 10/26/2012 - 06:26:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:28:33] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:30:59] 10/26/2012 - 06:30:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:36:00] 10/26/2012 - 06:36:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:41:00] 10/26/2012 - 06:41:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:41:56] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:43:32] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 142 processes [06:44:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:46:00] 10/26/2012 - 06:45:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:51:00] 10/26/2012 - 06:51:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:56:01] 10/26/2012 - 06:56:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [06:57:57] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.18, 4.18, 4.73 [07:01:05] 10/26/2012 - 07:01:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:06:08] 10/26/2012 - 07:06:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:11:07] 10/26/2012 - 07:11:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:14:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:16:02] 10/26/2012 - 07:16:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:21:00] 10/26/2012 - 07:21:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:21:56] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 2.78, 3.53, 4.98 [07:26:02] 10/26/2012 - 07:26:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:31:03] 10/26/2012 - 07:31:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:36:02] 10/26/2012 - 07:35:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:41:02] 10/26/2012 - 07:41:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:42:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:44:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:46:01] 10/26/2012 - 07:46:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:51:01] 10/26/2012 - 07:51:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [07:56:11] 10/26/2012 - 07:56:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:01:00] 10/26/2012 - 08:01:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:06:07] 10/26/2012 - 08:06:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:11:03] 10/26/2012 - 08:11:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:12:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:14:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:16:01] 10/26/2012 - 08:16:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:21:10] 10/26/2012 - 08:21:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:26:01] 10/26/2012 - 08:26:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:31:04] 10/26/2012 - 08:31:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:36:14] 10/26/2012 - 08:36:14 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:41:14] 10/26/2012 - 08:41:14 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:42:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:44:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:46:06] 10/26/2012 - 08:46:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:51:03] 10/26/2012 - 08:51:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [08:56:01] 10/26/2012 - 08:55:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:01:05] 10/26/2012 - 09:01:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:06:05] 10/26/2012 - 09:06:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:06:25] [bz] (8RESOLVED - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 38217] robots.txt - https://bugzilla.wikimedia.org/show_bug.cgi?id=38217 [09:11:00] 10/26/2012 - 09:11:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:12:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:14:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:16:02] 10/26/2012 - 09:16:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:20:59] 10/26/2012 - 09:20:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:26:05] 10/26/2012 - 09:26:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:31:08] 10/26/2012 - 09:31:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:36:02] 10/26/2012 - 09:36:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:41:10] 10/26/2012 - 09:41:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:42:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:44:26] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:46:01] 10/26/2012 - 09:46:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:51:07] 10/26/2012 - 09:51:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [09:56:11] 10/26/2012 - 09:56:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:01:08] 10/26/2012 - 10:01:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:06:01] 10/26/2012 - 10:06:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:11:01] 10/26/2012 - 10:11:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:12:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:14:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:15:58] 10/26/2012 - 10:15:58 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:21:00] 10/26/2012 - 10:21:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:26:00] 10/26/2012 - 10:26:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:31:02] 10/26/2012 - 10:31:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:36:06] 10/26/2012 - 10:36:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:41:09] 10/26/2012 - 10:41:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:43:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:44:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:45:32] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:45:42] PROBLEM Current Users is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:45:54] PROBLEM Total processes is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:45:54] PROBLEM Current Load is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:45:54] PROBLEM SSH is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Server answer: [10:46:01] 10/26/2012 - 10:46:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:46:32] PROBLEM Disk Space is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:49:52] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 42% free memory [10:50:37] RECOVERY dpkg-check is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: All packages OK [10:50:42] RECOVERY Current Users is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [10:50:52] RECOVERY Total processes is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: PROCS OK: 121 processes [10:50:52] RECOVERY Current Load is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK - load average: 0.06, 0.58, 0.63 [10:50:52] RECOVERY SSH is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [10:51:06] 10/26/2012 - 10:51:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:51:36] RECOVERY Disk Space is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: DISK OK [10:56:07] 10/26/2012 - 10:56:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:01:02] 10/26/2012 - 11:01:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:06:00] 10/26/2012 - 11:06:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:11:01] 10/26/2012 - 11:11:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:14:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:14:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:16:01] 10/26/2012 - 11:16:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:21:00] 10/26/2012 - 11:21:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:26:09] 10/26/2012 - 11:26:08 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:31:14] 10/26/2012 - 11:31:11 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:36:02] 10/26/2012 - 11:36:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:41:04] 10/26/2012 - 11:41:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:44:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:44:35] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:46:05] 10/26/2012 - 11:46:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:51:08] 10/26/2012 - 11:51:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:56:02] 10/26/2012 - 11:56:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:01:12] 10/26/2012 - 12:01:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:06:02] 10/26/2012 - 12:06:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:11:01] 10/26/2012 - 12:11:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:14:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:14:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:16:02] 10/26/2012 - 12:16:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:17:22] anyone knows why labs fails on Error: invalid magic word 'ISONLINE' [12:17:29] i.e. http://en.wikipedia.beta.wmflabs.org/wiki/File:Citygroup_Center,_San_Francisco.ogv [12:17:59] j^: It may be OnlineStatusBar, but I am not sure. [12:19:02] petan: Do you have any idea about this? [12:19:30] that is likely OSB issue, but I don't know why it throws that... [12:19:43] I think we can just disable the extension for now [12:20:08] looks disabled to me on the Gerrit repo, unless there is a live hack? [12:21:02] 10/26/2012 - 12:21:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:21:22] update was not working, fixed permissions. lets see if that fixes it [12:25:58] 10/26/2012 - 12:25:58 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:31:00] 10/26/2012 - 12:31:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:36:15] 10/26/2012 - 12:36:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:41:03] 10/26/2012 - 12:41:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:44:34] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:44:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:46:03] 10/26/2012 - 12:46:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:51:06] 10/26/2012 - 12:51:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:56:01] 10/26/2012 - 12:56:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:57:43] petan: OnlineStatusBar is enabled in wmf-config/ext-wmflabs.php [12:58:17] and its in git [12:59:12] yes [12:59:29] it was enabled for ages, so I have no idea why it started doing troubles now [12:59:43] I will disable it [13:01:12] 10/26/2012 - 13:01:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:05:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [13:06:01] 10/26/2012 - 13:06:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:10:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 103 processes [13:11:03] 10/26/2012 - 13:11:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:14:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:14:48] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:16:04] 10/26/2012 - 13:16:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:21:09] 10/26/2012 - 13:21:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:25:48] New patchset: jan; "Import LabsCentralauth-extension" [labs/centralauth] (extension) - https://gerrit.wikimedia.org/r/30152 [13:26:02] 10/26/2012 - 13:26:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:26:04] Change merged: jan; [labs/centralauth] (extension) - https://gerrit.wikimedia.org/r/30152 [13:27:46] Hydriz: I have developed a little extension for the CentralAuth-project. Can you review it please? http://centralauth.wmflabs.org/index.php/Special:LabsCentralauth [13:27:55] lol, I am in the midst of doing so [13:28:01] but you merged it :( [13:29:05] Hydriz: I want to autoupdate it and so I need a first version in Gerrit [13:29:38] okay, but what does the extension do? [13:30:16] It should download a Gerrit change for CentralAuth so it can be reviewed [13:30:55] Jan_Luca: Taking a look at it now, give me a moment. :) [13:31:01] 10/26/2012 - 13:31:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:31:31] but any idea why it loads quite slowly? [13:31:52] A moment please, I check the apache instance... [13:33:22] <^demon> Hydriz, Jan_Luca: http://www.mediawiki.org/wiki/How_to_debug#Profiling [13:33:41] ohai ^demon, was waiting for you :) [13:33:49] <^demon> Uh oh. [13:33:50] * ^demon hides [13:33:51] <^demon> :) [13:34:21] what a spam to GitHub with the extensions [13:34:21] ^demon: The labs instance seems to be slow [13:34:50] <^demon> Hydriz: They'll be replicating soon. Had to make them at some point. [13:35:05] ^demon: Thats good :) [13:36:05] 10/26/2012 - 13:36:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:36:50] !log centralauth Update centralauth-frontend and restart it [13:37:28] Why !log does not work? [13:38:20] Jan_Luca: Can I do the code review instead of just telling you on IRC? [13:38:24] It would be better with context. [13:38:51] Hydriz: Sure [13:38:55] thanks :) [13:39:35] PROBLEM dpkg-check is now: CRITICAL on centralauth-frontend i-00000458.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [13:41:02] 10/26/2012 - 13:41:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:44:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:45:26] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:46:19] 10/26/2012 - 13:46:19 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:51:01] 10/26/2012 - 13:51:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:52:28] New review: Hydriz; "Code review: Overall looks okay. Notes:" [labs/centralauth] (extension) - https://gerrit.wikimedia.org/r/30152 [13:52:43] Jan_Luca: Done :) [13:53:14] Hydriz: Thanks [13:53:28] np [13:53:29] <^demon> Jan_Luca: If I had known labs/centralauth was an extension, it should've gone in mediawiki/extensions/* probably :\ [13:53:38] ^demon: It isn't [13:53:50] <^demon> Sure looks like one... [13:53:59] Its just some config files in the project [13:54:14] but it wasn't my idea of putting very different files in different branches :) [13:56:01] 10/26/2012 - 13:56:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:57:19] ^demon: The extension is a special thing for this project [13:58:15] <^demon> There's lots of extensions that are special. Case in point: OpenStackManager :) [13:58:24] <^demon> Anyway, no need to bikeshed. [14:00:56] 10/26/2012 - 14:00:56 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:06:10] 10/26/2012 - 14:06:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:06:34] Any wikidata project members can you delete that directory in /home? Its very irritating [14:06:46] no, I take that back, its just spamming [14:11:04] 10/26/2012 - 14:11:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:14:48] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:15:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:16:04] 10/26/2012 - 14:16:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:21:05] 10/26/2012 - 14:21:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:26:01] 10/26/2012 - 14:26:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:31:02] 10/26/2012 - 14:31:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:36:00] 10/26/2012 - 14:36:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:41:03] 10/26/2012 - 14:41:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:45:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:46:01] 10/26/2012 - 14:46:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:46:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:51:06] 10/26/2012 - 14:51:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:54:33] Can someone review requests for project creation https://labsconsole.wikimedia.org/wiki/Requests#Request_for_project_creation ? [14:56:05] 10/26/2012 - 14:56:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:01:09] 10/26/2012 - 15:01:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [15:06:11] 10/26/2012 - 15:06:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:11:05] 10/26/2012 - 15:11:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:16:05] 10/26/2012 - 15:16:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:16:15] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:17:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:21:05] 10/26/2012 - 15:21:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:26:11] 10/26/2012 - 15:26:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:29:09] aude: I have now fixed the packages and updated the puppet scripts to move things out of the home directory [15:29:18] So things should mostly work now [15:29:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 103 processes [15:30:03] One thing I haven't figured out yet though is how to convince puppet to run the sql scripts in the correct order. So you need to run puppet 3 - 4 times for all the dependencies to be met. [15:31:01] 10/26/2012 - 15:31:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:36:02]