[00:35:35] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [00:38:15] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [00:39:55] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [00:39:55] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [00:40:33] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [00:46:16] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [00:57:55] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 14% free memory [01:07:55] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [01:20:34] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:55:35] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [03:09:15] PROBLEM Total processes is now: CRITICAL on deployment-squid.pmtpa.wmflabs 10.4.0.17 output: PROCS CRITICAL: 291 processes [03:19:23] RECOVERY Total processes is now: OK on deployment-squid.pmtpa.wmflabs 10.4.0.17 output: PROCS OK: 94 processes [03:32:24] PROBLEM Total processes is now: WARNING on deployment-squid.pmtpa.wmflabs 10.4.0.17 output: PROCS WARNING: 190 processes [04:31:26] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [04:37:55] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [04:37:55] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [04:41:14] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [04:41:24] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 22% free memory [04:49:15] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [04:55:47] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [05:00:56] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [05:04:26] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [05:50:33] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [05:55:35] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [06:32:56] PROBLEM Total processes is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS WARNING: 151 processes [06:34:16] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [06:47:55] RECOVERY Total processes is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS OK: 147 processes [08:05:25] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [08:39:14] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [08:40:23] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [08:40:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [09:05:35] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [09:07:15] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [09:08:25] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [09:10:33] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [09:13:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [09:28:23] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [09:55:36] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [10:06:24] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [10:30:36] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:20:34] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:29:15] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 28% free memory [12:19:27] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 162 processes [12:36:24] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [12:37:14] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [12:38:54] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [12:40:54] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [12:46:55] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 14% free memory [13:05:15] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [13:08:55] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [13:25:35] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [13:30:36] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [13:44:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 208 processes [14:09:24] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 189 processes [14:24:24] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 208 processes [14:25:25] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: CHECK_NRPE: Socket timeout after 10 seconds. [14:33:53] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [15:04:55] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [15:15:35] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [15:20:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [15:35:16] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [16:07:04] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [16:15:35] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [16:15:45] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 168 processes [16:17:55] PROBLEM Current Load is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: WARNING - load average: 17.87, 15.42, 9.40 [16:20:43] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 107 processes [16:32:54] RECOVERY Current Load is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: OK - load average: 0.00, 0.86, 3.76 [16:37:04] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [16:38:54] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [16:40:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [16:41:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:44:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 207 processes [16:45:36] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [16:46:56] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [16:59:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [17:07:05] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [17:09:25] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 193 processes [17:13:14] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [17:37:05] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [17:43:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [17:44:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 212 processes [18:02:24] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [18:07:34] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [18:09:24] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 195 processes [18:17:26] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [18:31:24] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [18:37:35] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [19:07:54] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [19:17:56] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 19% free memory [19:26:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 5.71, 5.71, 5.23 [19:36:45] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 1.12, 3.22, 4.46 [19:37:55] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [20:08:04] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [20:38:19] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [20:38:45] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [20:39:55] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [20:41:53] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [20:46:17] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [21:07:54] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [21:09:54] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [21:10:22] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [21:34:47] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 4.42, 6.71, 5.66 [21:34:57] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 153 processes [21:39:23] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.54, 9.09, 6.76 [21:39:43] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 2.82, 3.80, 4.66 [21:41:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.49, 9.61, 6.49 [21:42:03] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [21:59:55] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 149 processes [22:01:54] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 1.64, 2.25, 4.39 [22:04:24] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [22:04:24] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.63, 2.52, 4.48 [22:12:03] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [22:34:24] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 196 processes [22:42:32] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [23:12:33] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [23:42:33] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [23:54:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes