[00:01:54] 10/19/2012 - 00:01:53 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:06:04] 10/19/2012 - 00:06:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:11:01] 10/19/2012 - 00:11:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:14:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:16:05] 10/19/2012 - 00:16:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:21:13] 10/19/2012 - 00:21:13 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:24:08] !log deployment-prep csteipp: ran mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki en wikivoyage enwikivoyage en.voyage.beta.wmflabs.org [00:26:01] 10/19/2012 - 00:26:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:31:02] 10/19/2012 - 00:31:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:35:53] RECOVERY Current Load is now: OK on ve-roundtrip i-00000404.pmtpa.wmflabs output: OK - load average: 2.95, 3.93, 4.73 [00:36:00] 10/19/2012 - 00:36:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:41:03] 10/19/2012 - 00:40:55 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:44:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:46:09] 10/19/2012 - 00:46:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:51:02] 10/19/2012 - 00:50:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [00:56:09] 10/19/2012 - 00:56:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:01:04] 10/19/2012 - 01:01:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:05:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 172 processes [01:06:00] 10/19/2012 - 01:06:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:08:53] PROBLEM Current Load is now: WARNING on ve-roundtrip i-00000404.pmtpa.wmflabs output: WARNING - load average: 5.43, 6.04, 5.68 [01:10:53] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 95 processes [01:11:03] 10/19/2012 - 01:11:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:14:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:15:59] 10/19/2012 - 01:15:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:21:04] 10/19/2012 - 01:21:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:26:06] 10/19/2012 - 01:26:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:26:55] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.97, 3.25, 4.56 [01:31:20] 10/19/2012 - 01:31:18 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:36:03] 10/19/2012 - 01:36:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:40:58] 10/19/2012 - 01:40:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:46:04] 10/19/2012 - 01:46:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:46:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:51:10] 10/19/2012 - 01:51:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [01:56:01] 10/19/2012 - 01:55:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:01:19] 10/19/2012 - 02:01:19 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:06:02] 10/19/2012 - 02:06:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:10:59] 10/19/2012 - 02:10:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:12:28] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK WARNING - free space: /export 598 MB (3% inode=59%): /home/SAVE 598 MB (3% inode=59%): [02:16:09] 10/19/2012 - 02:16:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:17:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:21:10] 10/19/2012 - 02:21:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:26:06] 10/19/2012 - 02:26:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:31:09] 10/19/2012 - 02:31:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:36:11] 10/19/2012 - 02:36:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:41:00] 10/19/2012 - 02:40:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:46:02] 10/19/2012 - 02:46:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:49:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:50:59] 10/19/2012 - 02:50:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [02:56:12] 10/19/2012 - 02:56:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:01:23] 10/19/2012 - 03:01:22 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:06:08] 10/19/2012 - 03:06:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:11:06] 10/19/2012 - 03:11:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:16:09] 10/19/2012 - 03:16:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:21:03] 10/19/2012 - 03:21:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:26:05] 10/19/2012 - 03:26:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:31:17] 10/19/2012 - 03:31:14 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:36:06] 10/19/2012 - 03:36:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:41:00] 10/19/2012 - 03:40:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:46:04] 10/19/2012 - 03:46:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:49:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:51:03] 10/19/2012 - 03:51:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [03:55:59] 10/19/2012 - 03:55:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:01:04] 10/19/2012 - 04:01:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:06:02] 10/19/2012 - 04:06:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:11:10] 10/19/2012 - 04:11:07 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:14:52] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 15% free memory [04:16:08] 10/19/2012 - 04:16:07 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:21:13] 10/19/2012 - 04:21:12 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:25:59] 10/19/2012 - 04:25:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:31:12] 10/19/2012 - 04:31:11 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:36:13] 10/19/2012 - 04:36:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:39:55] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 4% free memory [04:41:11] 10/19/2012 - 04:41:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:44:55] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 96% free memory [04:46:09] 10/19/2012 - 04:46:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:49:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:51:02] 10/19/2012 - 04:51:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [04:55:57] 10/19/2012 - 04:55:57 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:01:22] 10/19/2012 - 05:01:21 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:06:00] 10/19/2012 - 05:05:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:11:12] 10/19/2012 - 05:11:11 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:12:22] RECOVERY Disk Space is now: OK on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK OK [05:16:05] 10/19/2012 - 05:16:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:19:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:20:59] 10/19/2012 - 05:20:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:26:05] 10/19/2012 - 05:26:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:31:08] 10/19/2012 - 05:31:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:36:13] 10/19/2012 - 05:36:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:41:01] 10/19/2012 - 05:41:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:46:01] 10/19/2012 - 05:46:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:49:35] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:51:08] 10/19/2012 - 05:51:07 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [05:56:00] 10/19/2012 - 05:55:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:01:19] 10/19/2012 - 06:01:19 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:06:14] 10/19/2012 - 06:06:14 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:11:07] 10/19/2012 - 06:11:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:16:26] 10/19/2012 - 06:16:25 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:19:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:21:27] 10/19/2012 - 06:21:22 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:26:42] 10/19/2012 - 06:26:42 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:31:09] 10/19/2012 - 06:31:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:36:12] 10/19/2012 - 06:36:11 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:41:30] 10/19/2012 - 06:41:27 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:45:15] Change on 12mediawiki a page Developer access was modified, changed by Nandor link https://www.mediawiki.org/w/index.php?diff=595387 edit summary: [06:46:15] 10/19/2012 - 06:46:15 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:49:34] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:51:09] 10/19/2012 - 06:51:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [06:56:35] 10/19/2012 - 06:56:35 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:01:05] 10/19/2012 - 07:01:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:06:11] 10/19/2012 - 07:06:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:11:10] 10/19/2012 - 07:11:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:16:02] 10/19/2012 - 07:16:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:19:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:21:04] 10/19/2012 - 07:21:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:26:02] 10/19/2012 - 07:26:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:31:06] 10/19/2012 - 07:31:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:35:13] PROBLEM dpkg-check is now: CRITICAL on bots-2 i-0000009c.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [07:36:19] 10/19/2012 - 07:36:19 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:40:34] RECOVERY dpkg-check is now: OK on bots-2 i-0000009c.pmtpa.wmflabs output: All packages OK [07:41:18] 10/19/2012 - 07:41:18 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:46:29] 10/19/2012 - 07:46:28 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:49:37] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:51:21] 10/19/2012 - 07:51:21 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [07:56:16] 10/19/2012 - 07:56:15 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:01:04] 10/19/2012 - 08:01:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:06:08] 10/19/2012 - 08:06:07 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:11:05] 10/19/2012 - 08:11:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:16:05] 10/19/2012 - 08:16:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:19:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:21:36] 10/19/2012 - 08:21:35 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:26:22] 10/19/2012 - 08:26:22 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:31:21] 10/19/2012 - 08:31:21 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:36:10] 10/19/2012 - 08:36:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:41:12] 10/19/2012 - 08:40:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:46:23] 10/19/2012 - 08:46:21 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:49:47] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:51:02] 10/19/2012 - 08:51:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [08:56:08] 10/19/2012 - 08:56:07 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:01:11] 10/19/2012 - 09:01:11 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:06:05] 10/19/2012 - 09:06:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:11:05] 10/19/2012 - 09:11:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:16:11] 10/19/2012 - 09:16:11 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:17:23] !ping [09:17:24] pong [09:17:29] !ping [09:17:29] pong [09:17:32] :o [09:17:37] slow... [09:19:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:20:49] Change on 12mediawiki a page Developer access was modified, changed by Federico-lox link https://www.mediawiki.org/w/index.php?diff=595461 edit summary: [09:21:09] 10/19/2012 - 09:21:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:26:19] 10/19/2012 - 09:26:15 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:31:12] 10/19/2012 - 09:31:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:36:20] 10/19/2012 - 09:36:20 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:41:05] 10/19/2012 - 09:41:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:46:20] 10/19/2012 - 09:46:19 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:49:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:51:10] 10/19/2012 - 09:51:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:56:20] 10/19/2012 - 09:56:20 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [09:58:52] RECOVERY Current Load is now: OK on ve-roundtrip i-00000404.pmtpa.wmflabs output: OK - load average: 3.84, 4.37, 4.78 [10:01:00] 10/19/2012 - 10:00:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:06:05] 10/19/2012 - 10:06:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:11:06] 10/19/2012 - 10:11:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:16:00] 10/19/2012 - 10:16:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:19:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:21:02] 10/19/2012 - 10:21:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:26:01] 10/19/2012 - 10:25:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:31:17] 10/19/2012 - 10:31:17 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:36:12] 10/19/2012 - 10:36:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:41:08] 10/19/2012 - 10:41:07 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:46:05] 10/19/2012 - 10:46:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:49:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:51:00] 10/19/2012 - 10:50:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:56:01] 10/19/2012 - 10:56:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [10:59:58] !log bots petrb: synced all bots application servers [11:01:08] 10/19/2012 - 11:01:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:06:12] 10/19/2012 - 11:06:12 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:08:26] !log bots petrb: moving logbot to bots-1 [11:08:55] !log bots petrb: test [11:11:02] 10/19/2012 - 11:11:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:13:53] !log blagh [11:13:53] Message missing. Nothing logged. [11:14:02] !log bots meh [11:15:07] !log bots bot is now on different server [11:15:07] bots is not a valid project. [11:15:12] o.o [11:15:19] !log blah [11:15:19] Message missing. Nothing logged. [11:16:02] 10/19/2012 - 11:16:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:16:25] !log bots test [11:16:25] Can't contact LDAP for project list. [11:16:25] bots is not a valid project. [11:19:01] mutante hey [11:19:09] there is some problem with LDAP [11:19:09] :/ [11:19:31] !log bots test [11:19:31] Can't contact LDAP for project list. [11:19:31] bots is not a valid project. [11:19:37] !log bots test [11:19:37] bots is not a valid project. [11:19:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:20:59] 10/19/2012 - 11:20:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:25:58] 10/19/2012 - 11:25:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:31:10] 10/19/2012 - 11:31:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:35:59] 10/19/2012 - 11:35:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:41:10] 10/19/2012 - 11:41:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:46:10] 10/19/2012 - 11:46:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:49:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:50:58] 10/19/2012 - 11:50:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [11:56:04] 10/19/2012 - 11:56:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:01:07] 10/19/2012 - 12:01:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:06:06] 10/19/2012 - 12:06:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:11:09] 10/19/2012 - 12:11:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:16:00] 10/19/2012 - 12:16:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:19:52] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:21:10] 10/19/2012 - 12:21:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:26:05] 10/19/2012 - 12:26:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:31:17] 10/19/2012 - 12:31:16 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:36:04] 10/19/2012 - 12:36:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:38:02] http://upload.beta.wmflabs.org/wikipedia/commons/1/1e/%C3%84delfors_%28Repperda%29-1534205.jpeg and all other uploaded files give a 403 Forbidden error right now [12:41:16] 10/19/2012 - 12:41:15 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:46:03] 10/19/2012 - 12:46:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:50:58] 10/19/2012 - 12:50:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [12:51:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:56:02] 10/19/2012 - 12:56:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:01:10] 10/19/2012 - 13:01:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:05:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [13:06:01] 10/19/2012 - 13:06:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:10:36] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [13:11:00] 10/19/2012 - 13:10:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:12:59] !log wikidata-dev wikidata-dev4: Installed puppetmaster::self to play. [13:12:59] Can't contact LDAP for project list. [13:12:59] wikidata-dev is not a valid project. [13:16:04] 10/19/2012 - 13:16:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:21:08] 10/19/2012 - 13:21:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:22:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:26:08] 10/19/2012 - 13:26:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:31:15] 10/19/2012 - 13:31:11 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:35:59] 10/19/2012 - 13:35:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:41:14] 10/19/2012 - 13:41:13 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:46:07] 10/19/2012 - 13:46:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:51:09] 10/19/2012 - 13:51:07 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [13:52:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:55:58] 10/19/2012 - 13:55:57 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:00:59] 10/19/2012 - 14:00:57 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:02:02] <^demon> Damianz: Someone should tell sugarcrm they're doing something wrong :\ [14:06:10] 10/19/2012 - 14:06:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:11:26] 10/19/2012 - 14:11:23 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:16:05] 10/19/2012 - 14:16:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:21:14] 10/19/2012 - 14:21:14 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:24:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:26:01] 10/19/2012 - 14:26:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:30:57] 10/19/2012 - 14:30:56 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:35:56] 10/19/2012 - 14:35:56 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:41:00] 10/19/2012 - 14:40:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:46:35] 10/19/2012 - 14:46:34 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:50:17] hashar: time for a couple beta questions? [14:50:19] csteipp: sure [14:50:22] morning:) [14:50:32] Good afternoon :) [14:50:49] ;-) [14:51:17] 10/19/2012 - 14:51:14 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:52:20] So, there are a lot of modified / untracked files in /home/wikipedia/common [14:52:41] indeed [14:52:42] How is the best way to get our merged wikivoyage config pull to that server? [14:52:45] I have opened a few bugs about it [14:52:52] let me find the tracking bug [14:52:58] Ah, great [14:53:24] here it is bug 41132 https://bugzilla.wikimedia.org/showdependencytree.cgi?id=41132&hide_resolved=0 [14:53:41] Reedy made a change for the all.dblist live hack [14:53:56] the way I have done merge so far is: [14:54:04] by stashing the current working copy [14:54:13] do a git pull [14:54:13] apply stashed changes [14:54:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:54:17] solve conflicts manually :-) [14:54:31] (aka restore wikiversions.dat , fix the php symlink to points back to php-master) [14:54:34] which is far from ideal [14:54:36] gatcha. Shouldn't be too bad [14:54:58] if your change just changed the php files, git pull should apply cleanly [14:54:58] Yeah, not ideal, but at least doable. [14:55:35] I was just starting to review operations/mediawiki-config :-] [14:55:39] so will come to Reedy change [14:55:50] The problem is we had one commit that added to all.dblist, we then did a commit to revert that, but I don't think I can get them to merge at once [14:56:00] 10/19/2012 - 14:55:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [14:56:58] Cool. But yeah, I think we'll have to stash-merge the commit that touched all.dblist. Not the end of the world.. [14:57:13] Second, is it safe to update squid's config and restart squid? [14:57:13] still require some additional manual step [14:57:20] reedy made a change to use something like all-wmflabs.dblist [14:57:29] Yep [14:57:29] so all.dblist will no more conflict :-] [14:57:48] That's the hope. We realized that all.dblist in production was used for centralauth :( [14:57:57] (after we merged the change) [14:58:19] ohh it s merged? [14:58:42] Reedy merged two changes in last night [14:59:04] ahh https://gerrit.wikimedia.org/r/#/c/28642/ [14:59:05] nice [14:59:44] yep, that one and the one from me and matthias: https://gerrit.wikimedia.org/r/#/c/28238/ [14:59:59] Which was the one that caused problems in production [15:00:11] [bz] (8RESOLVED - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 41133] beta all.dblist is a live hack - https://bugzilla.wikimedia.org/show_bug.cgi?id=41133 [15:00:14] oh, what happened, csteipp? [15:00:35] mlitn: apparently centralauth uses all.dblist [15:00:40] who new ;) [15:00:45] Chris most probably earned his """I broke the site, I am no more a rookie""" badge =] [15:00:46] s/new/knew/ [15:01:03] 10/19/2012 - 15:01:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:01:18] csteipp: so we just have an issue with centralauth ? [15:01:32] oh, so there actually _was_ something freaking out on the dblist change ^^ [15:02:14] After the second merge from Reedy, then it's all fixed up. But yeah, in production, centralauth loops through all db's in all.dblist, and searches for conflicting usernames when a user is created [15:02:40] So we broke user creation for about 30 minutes [15:03:10] csteipp: on beta you should fix your umask, you currently remove the group writable flag :-] [15:03:23] Oh shoot.. Sorry about that. [15:03:25] should be something like: umask 0002 in your ~/.bashrc [15:04:21] I am looking at the /home/wikipedia/common directory on dbdump [15:04:36] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [15:05:23] Shoot. Should I `chmod g+w *`? [15:05:40] csteipp: I did it :-] [15:05:53] you did? [15:06:01] I'm still seeing r-- [15:06:08] 10/19/2012 - 15:06:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:06:08] ohh [15:06:08] only did it in .git [15:06:13] so yeah go ahead :-] [15:06:24] that will take a while hehe [15:06:26] +1 for beta improvements, thanks [15:07:48] Alright, so before I jump on the train... just a sanity check. I think we still need to add voyage to the apache config in ops repo, right? [15:08:11] And then all should be working (in theory) [15:09:11] Or does beta have a different set of apache configs? [15:09:15] in theory :-] [15:09:23] beta has different apache files IIRC [15:09:45] I can't remember how they are configured :-( [15:10:31] No worries. Apache isn't too hard to reverse engineer :) [15:10:31] ahh [15:10:33] it is in /data/project/apache [15:10:37] which is a local git repository [15:10:48] Oh... ok, that's helpful to know [15:11:05] 10/19/2012 - 15:11:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:11:45] csteipp: will clean it up while you commute :-] [15:16:07] 10/19/2012 - 15:16:07 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:18:44] !log deployment-prep Cleaned up /data/project/apache directory a bit. It is a local git repository since beta has different apache configuration file than in production. Also removed from that git repo the common-local dir "git rm --cached common-local" [15:18:45] Can't contact LDAP for project list. [15:18:45] deployment-prep is not a valid project. [15:18:56] !log deployment-prep Cleaned up /data/project/apache directory a bit. It is a local git repository since beta has different apache configuration file than in production. Also removed from that git repo the common-local dir "git rm --cached common-local" [15:18:56] deployment-prep is not a valid project. [15:18:56] !log deployment-prep Cleaned up /data/project/apache directory a bit. It is a local git repository since beta has different apache configuration file than in production. Also removed from that git repo the common-local dir "git rm --cached common-local" [15:18:57] deployment-prep is not a valid project. [15:19:01] ok [15:19:13] that is the last time I have been using that bot [15:20:59] 10/19/2012 - 15:20:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:24:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:26:05] <^demon> aude: mediawiki/extensions/GoogleMaps is already in SVN so needs conversion, right? [15:26:14] 10/19/2012 - 15:26:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:26:48] <^demon> [[mw:Git/Conversion/Extensions still in svn]] says yes. Could you update that page instead, please? [15:29:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [15:31:11] 10/19/2012 - 15:31:10 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:35:43] ^demon: yes [15:35:59] 10/19/2012 - 15:35:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:40:59] 10/19/2012 - 15:40:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:46:04] 10/19/2012 - 15:46:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:51:07] 10/19/2012 - 15:51:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [15:54:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:56:11] 10/19/2012 - 15:56:11 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:01:05] 10/19/2012 - 16:01:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:06:10] 10/19/2012 - 16:06:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:07:31] aude: As I haven't (tried to) figured out how to use the labs/maps repository yet , I have pushed the puppet scripts I have so far to https://github.com/apmon/OSM-rendering-stack-deplou [16:07:37] I'll be adding the debian packages later on. [16:08:04] apmon_: that's helpful [16:10:24] * aude eager to get home and poke at them [16:11:06] 10/19/2012 - 16:11:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:11:25] let me know what you think once you get a chance to look at them [16:11:28] ok [16:12:04] i know that wmf folks are very eager to get this working, even if simple for now [16:12:54] great. [16:13:18] In a simple form, it shouldn't be too difficult. [16:14:09] agree :) [16:16:10] 10/19/2012 - 16:16:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:21:01] 10/19/2012 - 16:21:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:24:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:26:03] 10/19/2012 - 16:25:59 - User marktraceur may have been modified in LDAP or locally, updating key in project(s): phabricator,orgcharts,visualeditor [16:26:17] 10/19/2012 - 16:26:16 - Updating keys for marktraceur at /export/keys/marktraceur [16:26:52] PROBLEM Current Load is now: WARNING on ve-roundtrip i-00000404.pmtpa.wmflabs output: WARNING - load average: 10.27, 8.88, 6.08 [16:31:13] 10/19/2012 - 16:31:13 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:36:20] 10/19/2012 - 16:36:20 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:41:20] 10/19/2012 - 16:41:14 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:46:00] 10/19/2012 - 16:46:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:49:55] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.20, 7.33, 5.83 [16:51:15] 10/19/2012 - 16:51:15 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [16:54:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:55:59] 10/19/2012 - 16:55:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:01:05] 10/19/2012 - 17:01:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:06:15] 10/19/2012 - 17:06:15 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:11:16] 10/19/2012 - 17:11:13 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:16:00] 10/19/2012 - 17:16:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:21:03] 10/19/2012 - 17:21:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:24:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:25:38] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=595600 edit summary: /* User:Federico-lox */ [17:26:02] 10/19/2012 - 17:26:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:29:50] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=595604 edit summary: /* User:Nandor */ [17:31:13] 10/19/2012 - 17:31:13 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:36:17] 10/19/2012 - 17:36:16 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:41:06] 10/19/2012 - 17:41:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:46:11] 10/19/2012 - 17:46:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:50:10] Hey Reedy, you up for doing a merge of wmf-config in beta? [17:51:13] 10/19/2012 - 17:51:13 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [17:54:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:54:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.18, 3.22, 4.46 [17:56:08] 10/19/2012 - 17:56:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:01:31] 10/19/2012 - 18:01:28 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:05:58] 10/19/2012 - 18:05:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:06:53] RECOVERY Current Load is now: OK on ve-roundtrip i-00000404.pmtpa.wmflabs output: OK - load average: 4.72, 4.58, 4.90 [18:11:20] 10/19/2012 - 18:11:19 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:16:08] 10/19/2012 - 18:16:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:21:17] 10/19/2012 - 18:21:16 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:24:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:26:02] 10/19/2012 - 18:26:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:31:02] 10/19/2012 - 18:31:01 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:36:06] 10/19/2012 - 18:36:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:40:59] 10/19/2012 - 18:40:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:42:52] PROBLEM Total processes is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS WARNING: 154 processes [18:46:04] 10/19/2012 - 18:46:03 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:51:00] 10/19/2012 - 18:50:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:54:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:55:47] hi, is it possible to resize an instance? [18:56:12] we have a small (single-core) instance, and need more CPUs and a bit more RAM [18:56:16] 10/19/2012 - 18:56:16 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [18:56:26] project visualeditor, vm ve-nodejs [18:57:42] ram: 51200 [18:57:51] cores: 44 [18:57:58] these are the current quoatas for visualeditor project [18:58:12] i think i saw Ryan raise CPUs somewhere yesterday [18:58:27] but this is project-wide stuff [18:58:49] changing the instance type afaik its not possible [18:59:18] I failed to create another instance too, so maybe there is another quote we run against (the error messages give no clue, just 'error') [18:59:50] I was hoping that resizing instances would become possible after the openstack upgrade [19:00:05] pretty basic vm functionality after all ;) [19:00:46] maybe i am not up2date on this [19:01:05] * gwicke is spoiled by using OpenVZ, where you can adjust anything without even restarting instances [19:01:16] 10/19/2012 - 19:01:15 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:02:04] k, will try to rsync-clone then in the meantime [19:02:16] gwicke: hmm, so nothing but "error"? [19:02:24] on instance creation? [19:02:30] failed to create instance IIRC [19:02:52] RECOVERY Total processes is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS OK: 147 processes [19:03:05] Ryan had to check the log last time [19:03:49] apparently any kind of clues for users would make it too easy to debug things, so the errors are stashed away in the log.. [19:05:59] 10/19/2012 - 19:05:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:06:20] gwicke: let me try that, which instance type did you want [19:07:04] mutante: m1.xlarge [19:07:16] mainly for the cores, we don't need more storage [19:07:25] and not that much more memory [19:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 192 processes [19:10:51] grrr- public key root logins are disabled for some reason [19:11:19] 10/19/2012 - 19:11:18 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:11:46] gwicke: how did you want to name the instance? [19:12:03] mutante: do you mean the new or the old one? [19:12:14] the new one would have been parsoid-spof [19:12:30] the old one (to upgrade) is ve-nodejs [19:13:40] is there a way to enable root logins in a VM that is not reverted by puppet? [19:15:27] gwicke: confirmed there is an issue with "xlarge".. BUT.. i could create a "large" just fine [19:15:35] so it seems to be just the xlarge type [19:15:52] lets create a bug for that [19:16:04] 10/19/2012 - 19:16:00 - Created a home directory for dzahn in project(s): visualeditor [19:16:20] could be a fragmentation issue [19:16:38] (no machine with eight cores available) [19:17:17] do you want to keep this one i created now as parsoid-spof but just large [19:18:01] if the resizing is not possible, then I'd try to repurpose one of the eight-core machines for spof [19:18:15] and use the four-core machine for rt tests in the meantime [19:18:31] can do that myself though [19:18:52] re: the root logins. really need that? sudo -s is not enough? [19:19:29] a bit annoying if you want to rsync over another machine [19:19:52] either way, you need root for that [19:20:08] I also dislike entering my password [19:20:22] because anybody with access to the machine could easily sniff that [19:20:53] you also have storage that is shared between instances in the same project [19:21:12] 10/19/2012 - 19:21:09 - User dzahn may have been modified in LDAP or locally, updating key in project(s): visualeditor [19:21:32] mutante: the rsync is about cloning the system though, not just data [19:23:52] PROBLEM Current Load is now: CRITICAL on parsoid-spof i-000004d5.pmtpa.wmflabs output: Connection refused by host [19:26:05] 10/19/2012 - 19:25:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:27:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:28:56] RECOVERY Current Load is now: OK on parsoid-spof i-000004d5.pmtpa.wmflabs output: OK - load average: 0.05, 0.40, 0.35 [19:31:03] 10/19/2012 - 19:31:02 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:36:12] 10/19/2012 - 19:36:11 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:39:40] mutante: which is the mount point for project-wide storage? [19:40:09] so far we use our home dirs for that purpose.. [19:40:22] <^demon> /data/project [19:40:24] <^demon> iirc. [19:40:44] nothing in /data [19:40:52] it's automount [19:41:07] 10/19/2012 - 19:41:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:41:07] <^demon> That ^ [19:41:27] <^demon> Damianz: Can we shut sugarcrm up? [19:41:29] <^demon> Or something? [19:41:43] Damianz: how can I get it to mount? [19:41:45] I don't have access to the box it's on [19:41:53] gwicke: access it. [19:42:07] Damianz: a cd does not seem to work [19:42:08] cd /data/visualeditor [19:42:10] ^demon: Basically Ryan disabled removing dirs but left the message there so it just loops :( [19:42:22] gwicke: /data/project not visualeditor [19:42:25] ah [19:42:40] * Damianz pokes works vpn to let his macbook connect [19:42:40] thanks! [19:43:52] PROBLEM Current Load is now: CRITICAL on parsoid-spof i-000004d6.pmtpa.wmflabs output: Connection refused by host [19:44:09] ^demon: So the current solution for its bugs is 'it's going away as soon as move off nfs and use pam_makedir', though commenting out the msg line would stop the spam. [19:44:26] pooh pooh .. pooh .. [19:44:32] PROBLEM Current Users is now: CRITICAL on parsoid-spof i-000004d6.pmtpa.wmflabs output: Connection refused by host [19:44:49] sigh, if I retrieve XML data from the API, then that does not mean that I use an XML parser to get the data out of the XML .. [19:45:13] PROBLEM Disk Space is now: CRITICAL on parsoid-spof i-000004d6.pmtpa.wmflabs output: Connection refused by host [19:46:04] 10/19/2012 - 19:45:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:47:20] Beetstra: You could use regex if you wanted. [19:47:41] Damianz .. that was exactly what I did ... [19:48:21] Damianz: [19:48:22] * 2012-10-19 @ 19:23:28 - Somehow I retrieved an empty settings page for meta.wikimedia.org [19:48:22] * 2012-10-19 @ 19:23:36 - Somehow I retrieved an empty settings page for meta.wikimedia.org [19:48:22] * 2012-10-19 @ 19:23:36 - Somehow I retrieved an empty settings page for meta.wikimedia.org [19:48:53] RECOVERY Current Load is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: OK - load average: 0.13, 0.51, 0.36 [19:48:54] guess why [19:49:33] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [19:50:12] RECOVERY Disk Space is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: DISK OK [19:50:22] PROBLEM Current Users is now: WARNING on ve-nodejs i-00000245.pmtpa.wmflabs output: USERS WARNING - 7 users currently logged in [19:50:25] short tag? like [19:51:00] 10/19/2012 - 19:51:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:51:07] No .. the bot was doing perfect until somewhere this afternoon .. [19:51:52] .. that is, until the devs in their eternal wisdom .. added a tag to the XML output, which was not compatible with one of my regexes .. [19:55:26] not that I understand why '' needs 'contentformat' and 'xml:space' anyways, probably for those who do use that slow perl/pythin/php/whatever XML parser [19:56:06] 10/19/2012 - 19:56:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [19:56:45] That is why someone else wrote an xml parser for you. [19:57:07] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:57:22] if that makes that someone else happy, that is nice. I just ignore it, and regex the info that I need out of the api-output [19:59:40] good, bots are back running. I hate this type of things, you just don't know where things go wrong - especially if you did not change anything yourself. [20:00:44] sure there wasn't a changelog entry for it either :D [20:00:49] <3 json [20:01:19] 10/19/2012 - 20:01:13 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:02:34] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes [20:04:11] oh, for crying out loud [20:04:19] qaaa [20:04:21] It seems to be only on meta in certain circumstances [20:04:23] s/q/w/ [20:04:42] PROBLEM Current Load is now: WARNING on ve-nodejs i-00000245.pmtpa.wmflabs output: WARNING - load average: 6.49, 7.33, 5.90 [20:04:48] Beetstra: You know everyone just does /<.*iwant="YAY".*>/ really :D [20:05:03] http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=User:COIBot/Settings.css&rvprop=content&format=xml -> '' [20:05:26] http://meta.wikimedia.org/w/api.php?action=query&prop=revisions&titles=User:COIBot/Settings.css&rvprop=content&format=xml -> '' .... [20:06:01] 10/19/2012 - 20:05:58 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:06:01] and that contentformat was what fooled the bot ... [20:07:01] And that must be newly added in a patch this afternoon [20:07:22] http://meta.wikimedia.org/w/index.php?title=User:COIBot/LinkReports&curid=76207&diff=4292264&oldid=4292252 [20:07:46] (that was the last we heard of COIBot ..) [20:08:52] PROBLEM Current Load is now: CRITICAL on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: Connection refused by host [20:09:05] I have no idea what COIBot does but IMPLODE ALL THE THINGS [20:09:32] PROBLEM Current Users is now: CRITICAL on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: Connection refused by host [20:10:13] PROBLEM Disk Space is now: CRITICAL on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: Connection refused by host [20:10:35] COIBot is roughly the bot that the anti-spam effort on meta and en.wikipedia relies on [20:10:49] Together with COIBot's little helper: LiWa3 [20:11:03] PROBLEM Free ram is now: CRITICAL on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: Connection refused by host [20:11:04] 10/19/2012 - 20:11:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:13:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK - load average: 0.11, 0.57, 0.42 [20:14:37] RECOVERY Current Users is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [20:14:43] RECOVERY Current Load is now: OK on ve-nodejs i-00000245.pmtpa.wmflabs output: OK - load average: 2.97, 4.10, 4.96 [20:15:12] RECOVERY Disk Space is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: DISK OK [20:15:57] 10/19/2012 - 20:15:57 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:16:05] RECOVERY Free ram is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK: 3318% free memory [20:16:13] !log deployment-prep csteipp: ran git pull of master, kept local dblist conflicts [20:16:14] Can't contact LDAP for project list. [20:16:14] deployment-prep is not a valid project. [20:21:10] 10/19/2012 - 20:21:09 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:26:02] 10/19/2012 - 20:26:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:26:50] !log deployment-prep csteipp: ran git pull of master, kept local dblist conflicts [20:26:51] deployment-prep is not a valid project. [20:27:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [20:27:44] !log deployment-prep csteipp: ran git pull of master, kept local dblist conflicts [20:27:46] Logged the message, Master [20:28:15] * Damianz stops highlighting csteipp and goes to find f00d [20:29:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:30:57] I'd forgotten how much I hate svn [20:31:12] 10/19/2012 - 20:31:12 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:36:07] 10/19/2012 - 20:36:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:37:05] the vm storage seems to be quite slow currently- rsync copies a bigger file at ~320kB/s [20:37:21] from local vm storage to /data/project [20:41:00] 10/19/2012 - 20:40:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:46:08] 10/19/2012 - 20:46:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:46:08] yes [20:46:23] It's like a fs on an image on an fs on a disk. [20:46:52] ouch [20:51:06] 10/19/2012 - 20:51:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:56:14] 10/19/2012 - 20:56:12 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [20:57:04] Hey Reedy, do you know how to push out apache configs in beta? [20:57:21] magic [20:57:56] Dang, I think my wand is borked [20:58:19] it was symlinked to project storage at one point, not sure atm without looking [20:59:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:00:37] Yeah, it's now in /usr/local/apache/conf [21:01:07] 10/19/2012 - 21:01:06 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:06:16] 10/19/2012 - 21:06:16 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:09:47] Anyone know if scap-2 on beta is intentionally broken? Or if that's something I can fix? [21:11:09] 10/19/2012 - 21:11:08 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:15:26] RECOVERY Current Users is now: OK on ve-nodejs i-00000245.pmtpa.wmflabs output: USERS OK - 4 users currently logged in [21:16:01] 10/19/2012 - 21:16:00 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:21:18] 10/19/2012 - 21:21:14 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:26:18] 10/19/2012 - 21:26:14 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:29:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:30:59] 10/19/2012 - 21:30:56 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:36:16] 10/19/2012 - 21:36:16 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:41:05] 10/19/2012 - 21:41:05 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:46:04] 10/19/2012 - 21:46:04 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:51:00] 10/19/2012 - 21:50:59 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:53:21] yeah, sorry mutante just mentioned it to me [21:54:21] heh [21:54:35] yeah, it's because I left in the middle of setting some other stuff up [21:54:54] usually /var/lib/ is a better homedir for system users [21:54:54] and had to rush as soon as I unzipped something that happened to be in /home for no apparent reason ;) [21:54:54] ah [21:54:56] it's not actually a user... [21:54:56] I see [21:55:01] * Jamesofur fixes [21:55:09] thanks [21:55:57] 10/19/2012 - 21:55:57 - User SugarCE-Full-6.5.5 has been renamed, moving home directory in project(s): sugarcrm [21:57:03] Is it possible to rest a VM into the default state, or does one simply delete it and recreate? [21:57:42] apmon_: recreate [21:57:56] OK, thanks [21:57:59] yw [21:59:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:02:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes [22:06:35] !log sugarcrm sugar config files moved out of /home to /etc renamed to sugarcrm/ [22:06:36] Logged the message, Master [22:08:14] !log sugarcrm apache server stopped for config and setup… that should have been off anyway [22:12:36] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [22:25:14] connecting to an instance takes an eternity … [22:28:15] !log sugarcrm apache server started, permissions adjusted for configs [22:28:17] Logged the message, Master [22:29:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:42:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 103 processes [22:44:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 158 processes [22:58:52] PROBLEM Current Load is now: CRITICAL on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Connection refused by host [22:59:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:00:53] PROBLEM Total processes is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS WARNING: 158 processes [23:03:33] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [23:03:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 0.20, 0.58, 0.44 [23:07:36] our coordinator is now CPU-bound [23:07:52] oops, sorry- wrong channel [23:09:30] which is the recommended storage location for a database in a VM? [23:09:33] glusterfs seems to be very slow [23:29:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:40:56] RECOVERY Total processes is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS OK: 132 processes [23:50:25] !log sugarcrm test sugar setup and active [23:50:33] Logged the message, Master [23:57:12] PROBLEM Disk Space is now: CRITICAL on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [23:59:34] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)