[00:03:58] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [00:11:51] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [00:34:05] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [00:41:55] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [01:04:21] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [01:08:15] can anyone think of a labs host that has an HTTPS server installed already? [01:09:49] Emw: GChriss: i chatted with Ryan_Lane1 earlier today but forgot to ask him [01:12:41] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [01:19:50] jeremyb: many thanks. It's not especially urgent, but I would enjoy trying my hand at migrating over the codebase (and knowing it is in a "safe" location) [01:20:50] ohhhhhh [01:20:56] you want a gerrit project then? [01:21:02] i was thinking labs project [01:21:06] both? [01:21:08] gerrit is a git host [01:21:23] right -- I've read through almost all of gerrit documentation [01:21:57] sad that i have the group id memorized, 119 ;) [01:22:00] https://gerrit.wikimedia.org/r/#/admin/groups/119,info [01:23:23] as far as I can tell metavidwiki isn't in the gerrit trunk/extensions branch (or anywhere else); I've made a number of changes since the initial metavidwiki checkout 3 yrs ago. so, if I understand correctly, now would be a good time to copy metavidwiki from svn into gerrit, run a 'diff' locally through the source tree, and submit patches through gerrit [01:23:27] robla's /away but also looks fairly unidle [01:23:54] uhhh, i guess [01:24:13] independently of that I'm looking forward to figuring out puppet + setting up a dev-only version of openmeetings.org [01:24:15] you might want to do one massive code review unless you already have it broken into individual patches [01:24:16] in labs [01:25:11] I don't have individual patches ready, an en-mass code review would be fine by me, and there are a ton of changes [01:25:45] but the changes I have hacked are important enough to identify/commit [01:25:47] note you don't need any review for a labs deployment [01:26:01] right, just a project :) [01:26:34] the sister projects proposal is looking better: http://meta.wikimedia.org/wiki/OpenMeetings.org [01:27:20] you could also have it on gerrit without review at all and then over time you can move stuff from the huge diff into a separate gerrit branch that does have code review. and things can be removed from the mega patch over time as they make their way into individual changes [01:28:31] do you mean 'code review' in a security sense? [01:30:02] also in a this doesn't break the site or cause dataloss or do something unintended sense [01:31:05] Thehelpfulone: ping [01:31:34] oh, pharos is on the committee too [01:31:39] i guess Thehelpfulone is asleep [01:34:25] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [01:35:45] * robla looks up... [01:35:48] hey! [01:36:00] I'm here for a sec...what's up? [01:36:02] gerrit project creation and labs project creation [01:36:14] i think labs is more of a priority [01:36:23] gerrit can wait for demon to do a repo conversion [01:37:20] I'm not the guy to go to for labs project creation [01:37:29] :( [01:37:52] who're the people to bug about that? [01:37:55] Anyone know what could be causing this error? http://integration.wmflabs.org/mw/ [01:38:01] I can't get mw to run on sqlite there [01:38:03] * jeremyb was just pulling up https://labsconsole.wikimedia.org/wiki/Special:ListUsers/sysop ;) [01:38:14] permissions are fine (tried 777 even) [01:38:19] libraries are installed [01:38:31] (Can't contact the database server: Cannot return last error, no db connection) [01:38:35] jeremyb: yup, that looks like the list [01:38:53] * robla notes that he's not on it [01:38:59] robla: WikiSysop looks sketchy [01:39:10] "root" [01:39:52] robla: sorry, i asked you because you were on the gerrit list and then realized we want a repo conversion [01:40:12] Krinkle: is the database server pingable from that machine? [01:40:19] no prob....I didn't mean to sound cranky about being pinged [01:40:24] Emw: sqlite.. [01:40:32] Emw: in the same directory [01:40:33] robla: nah ;) [01:40:56] Emw: from maintenance/eval.php things like wfMsg() and wfGetDB() work fine [01:40:57] Emw: it's not a network thing nor a server thing [01:41:18] (well server as in box. but not service) [01:41:25] ah, ok [01:42:45] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:04:36] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [02:12:46] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:24:47] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [02:32:57] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 5.70, 5.67, 5.18 [02:34:37] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [02:34:57] RECOVERY Current Users is now: OK on bastion-restricted1 i-0000019b output: USERS OK - 5 users currently logged in [02:39:26] PROBLEM Total Processes is now: CRITICAL on ganglia-test2 i-00000250 output: PROCS CRITICAL: 212 processes [02:40:10] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 25% free memory [02:44:30] PROBLEM Total Processes is now: WARNING on ganglia-test2 i-00000250 output: PROCS WARNING: 185 processes [02:44:35] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:57:49] PROBLEM Free ram is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:58:37] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:58:37] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:01:13] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [03:01:13] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:02:22] PROBLEM Free ram is now: UNKNOWN on gluster-4 i-000002e4 output: NRPE: Unable to read output [03:03:28] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 94% free memory [03:03:28] RECOVERY Total Processes is now: OK on worker1 i-00000208 output: PROCS OK: 75 processes [03:06:08] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 10% free memory [03:06:08] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [03:08:35] PROBLEM Free ram is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:10:21] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 69 MB (5% inode=57%): [03:12:00] PROBLEM Total Processes is now: CRITICAL on ganglia-test2 i-00000250 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:12:11] PROBLEM Free ram is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [03:14:00] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 6.20, 5.22, 4.10 [03:15:26] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [03:16:48] PROBLEM Free ram is now: UNKNOWN on configtest-main i-000002dd output: NRPE: Unable to read output [03:17:58] PROBLEM Free ram is now: UNKNOWN on psm-precise i-000002f2 output: NRPE: Unable to read output [03:23:55] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 1.95, 2.83, 3.51 [03:28:55] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.90, 1.52, 2.73 [03:34:55] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [03:36:36] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:36:55] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [03:39:55] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 15% free memory [03:46:28] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [03:50:20] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [03:57:01] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [03:59:52] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 2% free memory [04:02:09] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:05:40] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:06:14] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [04:06:15] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [04:06:31] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:07:18] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [04:11:20] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 5.81, 6.59, 6.33 [04:11:20] RECOVERY dpkg-check is now: OK on incubator-bot2 i-00000252 output: All packages OK [04:16:00] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:17:22] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [04:20:22] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 14% free memory [04:25:58] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:37:56] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [04:40:34] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:45:23] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:47:53] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [04:52:15] PROBLEM Total Processes is now: WARNING on ganglia-test2 i-00000250 output: PROCS WARNING: 187 processes [05:01:08] PROBLEM Current Users is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [05:01:08] PROBLEM dpkg-check is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [05:01:08] PROBLEM Disk Space is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [05:01:08] PROBLEM Current Load is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [05:01:08] PROBLEM Total Processes is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [05:01:13] PROBLEM Free ram is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [05:05:53] RECOVERY Current Users is now: OK on mwreview i-000002ae output: USERS OK - 0 users currently logged in [05:05:53] RECOVERY Disk Space is now: OK on mwreview i-000002ae output: DISK OK [05:05:53] RECOVERY dpkg-check is now: OK on mwreview i-000002ae output: All packages OK [05:05:53] RECOVERY Current Load is now: OK on mwreview i-000002ae output: OK - load average: 0.37, 2.13, 1.88 [05:05:53] RECOVERY Total Processes is now: OK on mwreview i-000002ae output: PROCS OK: 108 processes [05:05:58] RECOVERY Free ram is now: OK on mwreview i-000002ae output: OK: 68% free memory [05:08:43] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [05:18:54] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [05:38:56] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [05:45:34] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [05:48:57] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [05:51:00] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 69 MB (5% inode=57%): [06:09:05] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [06:19:01] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [06:21:11] PROBLEM Free ram is now: UNKNOWN on gluster-4 i-000002e4 output: NRPE: Unable to read output [06:27:23] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:31:31] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 8.16, 7.75, 6.92 [06:37:03] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:42:02] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [06:42:19] PROBLEM Free ram is now: CRITICAL on wikistats-history-01 i-000002e2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:35] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 18% free memory [06:46:19] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:46:30] PROBLEM SSH is now: CRITICAL on bots-sql2 i-000000af output: CRITICAL - Socket timeout after 10 seconds [06:48:29] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [06:48:29] PROBLEM Total Processes is now: CRITICAL on ganglia-test2 i-00000250 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:48:35] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:35] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [06:52:51] PROBLEM Free ram is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:56] PROBLEM Current Load is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:56] PROBLEM Disk Space is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:56] PROBLEM Current Users is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:56] PROBLEM Total Processes is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:59:09] PROBLEM Disk Space is now: CRITICAL on ipv6test1 i-00000282 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:59:19] PROBLEM Free ram is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:12] PROBLEM Current Load is now: WARNING on pybal-precise i-00000289 output: WARNING - load average: 6.37, 6.67, 5.36 [07:06:28] PROBLEM Free ram is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [07:07:56] PROBLEM Total Processes is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:01] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:01] PROBLEM Current Users is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:02] PROBLEM Free ram is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:02] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:02] PROBLEM Disk Space is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:10:03] RECOVERY SSH is now: OK on bots-sql2 i-000000af output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [07:11:31] PROBLEM HTTP is now: CRITICAL on integration-apache1 i-000002eb output: Connection refused [07:11:31] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 8% free memory [07:11:31] RECOVERY Current Users is now: OK on integration-apache1 i-000002eb output: USERS OK - 0 users currently logged in [07:11:36] PROBLEM Current Load is now: WARNING on incubator-bot2 i-00000252 output: WARNING - load average: 4.04, 5.49, 6.97 [07:11:36] RECOVERY Current Users is now: OK on incubator-bot2 i-00000252 output: USERS OK - 0 users currently logged in [07:11:36] RECOVERY Disk Space is now: OK on incubator-bot2 i-00000252 output: DISK OK [07:11:36] RECOVERY Free ram is now: OK on incubator-bot2 i-00000252 output: OK: 36% free memory [07:11:41] PROBLEM Current Load is now: CRITICAL on grail i-000002c6 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:40] PROBLEM dpkg-check is now: CRITICAL on ganglia-test2 i-00000250 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:34] PROBLEM Total Processes is now: CRITICAL on grail i-000002c6 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:57] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:57] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:57] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:58] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:58] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:19] RECOVERY dpkg-check is now: OK on incubator-bot2 i-00000252 output: All packages OK [07:14:57] PROBLEM Current Load is now: WARNING on ganglia-test2 i-00000250 output: WARNING - load average: 7.31, 7.43, 6.52 [07:15:13] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [07:15:13] PROBLEM Current Load is now: WARNING on upload-wizard i-0000021c output: WARNING - load average: 7.26, 6.14, 5.21 [07:15:13] RECOVERY Disk Space is now: OK on upload-wizard i-0000021c output: DISK OK [07:15:13] RECOVERY Total Processes is now: OK on upload-wizard i-0000021c output: PROCS OK: 91 processes [07:15:18] RECOVERY Free ram is now: OK on upload-wizard i-0000021c output: OK: 93% free memory [07:15:18] RECOVERY Current Users is now: OK on upload-wizard i-0000021c output: USERS OK - 0 users currently logged in [07:15:43] PROBLEM dpkg-check is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:15:43] PROBLEM Free ram is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:16:38] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 70 MB (5% inode=57%): [07:16:48] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 6.54, 9.90, 16.69 [07:16:48] PROBLEM Free ram is now: UNKNOWN on integration-apache1 i-000002eb output: NRPE: Unable to read output [07:16:48] RECOVERY Current Load is now: OK on integration-apache1 i-000002eb output: OK - load average: 1.31, 3.83, 4.72 [07:16:53] PROBLEM Free ram is now: CRITICAL on ganglia-test2 i-00000250 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:29] PROBLEM Current Load is now: WARNING on configtest-main i-000002dd output: WARNING - load average: 8.75, 8.65, 6.74 [07:18:29] PROBLEM Free ram is now: UNKNOWN on gluster-4 i-000002e4 output: NRPE: Unable to read output [07:18:29] PROBLEM Free ram is now: UNKNOWN on configtest-main i-000002dd output: NRPE: Unable to read output [07:18:29] PROBLEM Current Load is now: WARNING on grail i-000002c6 output: WARNING - load average: 3.55, 5.05, 5.98 [07:18:29] RECOVERY Total Processes is now: OK on grail i-000002c6 output: PROCS OK: 105 processes [07:18:34] RECOVERY HTTP is now: OK on integration-apache1 i-000002eb output: HTTP OK: HTTP/1.1 200 OK - 1104 bytes in 2.449 second response time [07:18:34] PROBLEM Total Processes is now: CRITICAL on bots-cb i-0000009e output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:18] PROBLEM Current Load is now: WARNING on gluster-4 i-000002e4 output: WARNING - load average: 5.25, 6.30, 6.13 [07:19:18] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:19] PROBLEM Current Load is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:19] PROBLEM Disk Space is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:19] PROBLEM Current Users is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:19] PROBLEM dpkg-check is now: CRITICAL on bots-cb i-0000009e output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:19] PROBLEM Total Processes is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:41] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:42] PROBLEM Current Load is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:42] PROBLEM Total Processes is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:23:39] PROBLEM Current Load is now: WARNING on precise-test i-00000231 output: WARNING - load average: 8.31, 6.53, 5.76 [07:23:39] PROBLEM Free ram is now: UNKNOWN on testforx i-000002f3 output: NRPE: Unable to read output [07:23:39] PROBLEM Current Load is now: WARNING on maps-tilemill1 i-00000294 output: WARNING - load average: 6.12, 7.76, 6.59 [07:23:44] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [07:23:44] PROBLEM Current Load is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:23:54] PROBLEM Current Load is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:24:05] PROBLEM Current Load is now: CRITICAL on ganglia-test2 i-00000250 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:26:43] PROBLEM host: psm-precise is DOWN address: i-000002f2 PING CRITICAL - Packet loss = 100% [07:26:43] PROBLEM Free ram is now: WARNING on ganglia-test2 i-00000250 output: Warning: 19% free memory [07:26:43] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 5.68, 7.22, 7.10 [07:26:43] RECOVERY Total Processes is now: OK on bots-cb i-0000009e output: PROCS OK: 113 processes [07:26:48] RECOVERY Total Processes is now: OK on mobile-testing i-00000271 output: PROCS OK: 132 processes [07:26:53] PROBLEM Current Load is now: WARNING on rds i-00000207 output: WARNING - load average: 4.63, 6.31, 6.06 [07:26:53] RECOVERY dpkg-check is now: OK on bots-cb i-0000009e output: All packages OK [07:27:03] PROBLEM Current Load is now: WARNING on reportcard2 i-000001ea output: WARNING - load average: 4.38, 5.42, 5.91 [07:27:03] RECOVERY Total Processes is now: OK on reportcard2 i-000001ea output: PROCS OK: 92 processes [07:27:08] RECOVERY Disk Space is now: OK on reportcard2 i-000001ea output: DISK OK [07:27:08] RECOVERY Current Users is now: OK on reportcard2 i-000001ea output: USERS OK - 0 users currently logged in [07:27:15] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:27:20] PROBLEM Current Load is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:28:37] PROBLEM Total Processes is now: WARNING on ganglia-test2 i-00000250 output: PROCS WARNING: 197 processes [07:29:27] PROBLEM Free ram is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:29:27] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:29:27] PROBLEM Current Load is now: CRITICAL on grail i-000002c6 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:29:37] PROBLEM Current Load is now: CRITICAL on hugglewiki i-000000aa output: CHECK_NRPE: Socket timeout after 10 seconds. [07:29:47] RECOVERY host: psm-precise is UP address: i-000002f2 PING OK - Packet loss = 0%, RTA = 0.54 ms [07:31:54] lo [07:33:09] RECOVERY Total Processes is now: OK on incubator-bot2 i-00000252 output: PROCS OK: 141 processes [07:33:32] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:34:38] PROBLEM Current Load is now: CRITICAL on bots-cb i-0000009e output: CRITICAL - load average: 105.48, 50.95, 25.70 [07:34:38] RECOVERY Current Load is now: OK on gluster-4 i-000002e4 output: OK - load average: 0.08, 1.19, 3.33 [07:35:03] RECOVERY Current Load is now: OK on worker1 i-00000208 output: OK - load average: 5.35, 5.31, 4.98 [07:35:03] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [07:35:03] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [07:35:03] RECOVERY Total Processes is now: OK on worker1 i-00000208 output: PROCS OK: 90 processes [07:35:08] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 94% free memory [07:35:08] PROBLEM Current Load is now: WARNING on mwreview i-000002ae output: WARNING - load average: 5.74, 7.69, 7.47 [07:35:08] PROBLEM Current Load is now: WARNING on hugglewiki i-000000aa output: WARNING - load average: 2.35, 5.34, 6.36 [07:35:09] PROBLEM Current Load is now: WARNING on pybal-precise i-00000289 output: WARNING - load average: 6.60, 6.23, 6.29 [07:35:09] PROBLEM Current Load is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:09] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:14] PROBLEM Current Load is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:24] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:36:04] PROBLEM Free ram is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [07:36:53] PROBLEM Current Load is now: WARNING on precise-test i-00000231 output: WARNING - load average: 3.65, 5.22, 5.60 [07:36:53] PROBLEM Current Load is now: WARNING on ganglia-test2 i-00000250 output: WARNING - load average: 5.03, 5.01, 5.74 [07:36:58] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:31] PROBLEM Current Load is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:36] PROBLEM Current Users is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:36] PROBLEM Disk Space is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:37] PROBLEM Total Processes is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:47] PROBLEM Free ram is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:36] PROBLEM Current Users is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:36] PROBLEM Total Processes is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:41] PROBLEM Current Users is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:41] PROBLEM Disk Space is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:41] PROBLEM dpkg-check is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:41] PROBLEM Free ram is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:13] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 1.90, 19.82, 19.32 [07:40:14] PROBLEM Free ram is now: CRITICAL on ganglia-test2 i-00000250 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:14] PROBLEM Current Load is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:14] PROBLEM Total Processes is now: CRITICAL on ganglia-test2 i-00000250 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:33] RECOVERY Current Load is now: OK on hugglewiki i-000000aa output: OK - load average: 0.19, 1.79, 4.43 [07:40:49] PROBLEM Total Processes is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:54] PROBLEM Current Load is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:54] PROBLEM Free ram is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:54] PROBLEM dpkg-check is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:54] PROBLEM Disk Space is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:58] RECOVERY Current Load is now: OK on maps-tilemill1 i-00000294 output: OK - load average: 4.53, 4.83, 4.89 [07:42:09] RECOVERY Free ram is now: OK on reportcard2 i-000001ea output: OK: 88% free memory [07:42:09] RECOVERY dpkg-check is now: OK on reportcard2 i-000001ea output: All packages OK [07:42:57] RECOVERY Current Load is now: OK on migration1 i-00000261 output: OK - load average: 1.47, 3.94, 4.73 [07:44:07] RECOVERY Current Users is now: OK on precise-test i-00000231 output: USERS OK - 0 users currently logged in [07:44:07] RECOVERY Total Processes is now: OK on mwreview i-000002ae output: PROCS OK: 108 processes [07:44:12] RECOVERY Current Users is now: OK on mwreview i-000002ae output: USERS OK - 0 users currently logged in [07:44:12] RECOVERY Disk Space is now: OK on mwreview i-000002ae output: DISK OK [07:44:12] RECOVERY Free ram is now: OK on mwreview i-000002ae output: OK: 68% free memory [07:44:12] RECOVERY dpkg-check is now: OK on mwreview i-000002ae output: All packages OK [07:45:12] PROBLEM Current Load is now: WARNING on configtest-main i-000002dd output: WARNING - load average: 0.80, 4.61, 6.19 [07:45:18] RECOVERY dpkg-check is now: OK on ganglia-test2 i-00000250 output: All packages OK [07:45:18] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:45:56] RECOVERY Total Processes is now: OK on migration1 i-00000261 output: PROCS OK: 91 processes [07:46:01] RECOVERY Free ram is now: OK on migration1 i-00000261 output: OK: 89% free memory [07:46:01] RECOVERY Disk Space is now: OK on migration1 i-00000261 output: DISK OK [07:46:01] RECOVERY dpkg-check is now: OK on migration1 i-00000261 output: All packages OK [07:46:30] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [07:55:03] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [08:01:33] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 0.56, 1.25, 6.42 [08:11:27] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 3.57, 1.69, 3.96 [08:16:51] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [08:20:25] PROBLEM Free ram is now: UNKNOWN on gluster-4 i-000002e4 output: NRPE: Unable to read output [08:20:25] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 8% free memory [08:21:35] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 1.99, 1.63, 2.84 [08:22:05] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [08:24:05] PROBLEM Free ram is now: UNKNOWN on psm-precise i-000002f2 output: NRPE: Unable to read output [08:26:04] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [08:42:14] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [08:47:14] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [08:56:36] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [08:58:36] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 16% free memory [09:01:08] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [09:04:14] PROBLEM Current Users is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [09:04:14] PROBLEM Disk Space is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [09:04:14] PROBLEM Free ram is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [09:04:14] PROBLEM Total Processes is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [09:04:19] PROBLEM dpkg-check is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [09:11:05] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 5.36, 6.20, 6.13 [09:15:56] PROBLEM Free ram is now: UNKNOWN on configtest-main i-000002dd output: NRPE: Unable to read output [09:17:08] !deployment-prep setting up a bit cache using varnish puppet classes. [09:17:08] deployment-prep is a project to test mediawiki at beta.wmflabs.org before putting it to prod [09:18:36] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [09:21:03] PROBLEM Free ram is now: WARNING on ganglia-test2 i-00000250 output: Warning: 17% free memory [09:21:03] PROBLEM Total Processes is now: WARNING on ganglia-test2 i-00000250 output: PROCS WARNING: 188 processes [09:26:57] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [09:27:58] OHHHH [09:28:01] varnish started \O/ [09:49:40] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [09:59:54] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [10:06:33] 06/28/2012 - 10:06:32 - Creating a project directory for pdbhandler [10:06:33] 06/28/2012 - 10:06:32 - Created a home directory for dzahn in project(s): pdbhandler [10:07:32] 06/28/2012 - 10:07:31 - User dzahn may have been modified in LDAP or locally, updating key in project(s): pdbhandler [10:08:54] PROBLEM Free ram is now: CRITICAL on test3 i-00000093 output: Critical: 2% free memory [10:12:33] 06/28/2012 - 10:12:33 - Created a home directory for emw in project(s): pdbhandler [10:13:33] 06/28/2012 - 10:13:33 - User emw may have been modified in LDAP or locally, updating key in project(s): pdbhandler [10:13:34] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [10:20:34] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [10:30:57] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [10:38:48] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [10:40:58] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 6% free memory [10:43:38] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [10:51:40] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [10:59:31] !log deployment-prep set group write on deployment-nfs-memc:/mnt/export/apache/common-local would let us rewrite the wikiversions.cdb file [10:59:32] Logged the message, Master [11:01:20] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [11:03:53] !log deployment-prep migrated all wiki from php-trunk to php-master by editing wikiversions.dat. Refreshed wikiversions.cdb and renamed ExtensionMessages-trunk.php to ExtensionMessages-master.php [11:03:55] Logged the message, Master [11:21:50] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [11:26:10] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 4.37, 4.67, 4.99 [11:31:53] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [11:34:13] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 4.46, 4.92, 5.01 [11:39:23] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 4.91, 4.82, 4.93 [11:51:54] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [12:01:54] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [12:16:20] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: Critical: 5% free memory [12:22:00] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [12:31:31] PROBLEM Total Processes is now: CRITICAL on ganglia-test2 i-00000250 output: PROCS CRITICAL: 205 processes [12:31:36] RECOVERY Free ram is now: OK on ganglia-test2 i-00000250 output: OK: 86% free memory [12:32:00] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [12:36:24] PROBLEM Total Processes is now: WARNING on ganglia-test2 i-00000250 output: PROCS WARNING: 191 processes [12:51:04] !log pdbhandler this is a new project dedicated to '"a MediaWiki extension to enable interactive 3D models of large molecules, like proteins and DNA', owner: Emw [12:51:05] Logged the message, Master [12:52:04] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [13:06:24] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [13:22:04] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [13:27:14] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 6.28, 6.35, 5.99 [13:33:34] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [13:36:48] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [13:41:44] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 70 MB (5% inode=57%): [13:52:43] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [14:01:22] RECOVERY Puppet freshness is now: OK on deployment-imagescaler01 i-0000025a output: puppet ran at Thu Jun 28 14:00:54 UTC 2012 [14:07:03] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [14:10:29] !log deployment-prep running puppet on apache{30,31}. The /etc/sudoers conflict has been merged in :) [14:10:31] Logged the message, Master [14:13:00] notice: /Stage[main]/Base::Vimconfig/File[/etc/alternatives/editor]/target: target changed '/bin/nano' to '/usr/bin/vim' [14:13:02] ... [14:15:17] RECOVERY Puppet freshness is now: OK on deployment-apache30 i-000002d3 output: puppet ran at Thu Jun 28 14:15:12 UTC 2012 [14:16:37] RECOVERY Puppet freshness is now: OK on deployment-jobrunner05 i-0000028c output: puppet ran at Thu Jun 28 14:16:23 UTC 2012 [14:20:33] 06/28/2012 - 14:20:32 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [14:23:23] PROBLEM dpkg-check is now: CRITICAL on deployment-apache31 i-000002d4 output: DPKG CRITICAL dpkg reports broken packages [14:23:38] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [14:23:55] mhmm [14:24:05] looks like I broke nrpe [14:24:21] PROBLEM Total Processes is now: CRITICAL on deployment-apache30 i-000002d3 output: Connection refused by host [14:24:26] PROBLEM Free ram is now: CRITICAL on deployment-apache30 i-000002d3 output: Connection refused by host [14:25:09] you did? [14:25:40] paravoid: did the cabling problem ever get fixed? [14:25:55] not as far as I know [14:26:02] :( [14:26:25] yeah... [14:28:19] ATTENTION!! rebooting maganese (aka gerrit) for dist-upgrade [14:29:18] RECOVERY Total Processes is now: OK on deployment-apache30 i-000002d3 output: PROCS OK: 127 processes [14:29:23] RECOVERY Free ram is now: OK on deployment-apache30 i-000002d3 output: OK: 92% free memory [14:32:47] Jeff_Green: cool! RT-1802 ? [14:34:19] mutante: formey? [14:34:33] eh, yeah, it says gerrit [14:34:43] did gerrit move after this ticket was created? the hostname is manganese [14:35:17] gerrit used to be on formey [14:35:21]