[00:02:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [00:33:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [00:33:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 190 processes [00:53:44] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [01:03:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [01:11:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 289 processes [01:16:42] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 153 processes [01:21:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 99 processes [01:33:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [02:03:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 190 processes [02:04:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [02:34:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [02:49:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 220 processes [03:04:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [03:09:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 181 processes [03:23:53] PROBLEM Current Load is now: CRITICAL on mobile-varnish.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [03:24:33] PROBLEM Disk Space is now: CRITICAL on mobile-varnish.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [03:25:13] PROBLEM Free ram is now: CRITICAL on mobile-varnish.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [03:26:43] PROBLEM Total processes is now: CRITICAL on mobile-varnish.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [03:27:23] PROBLEM dpkg-check is now: CRITICAL on mobile-varnish.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [03:34:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [04:04:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 205 processes [04:06:52] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [04:24:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 167 processes [04:27:33] PROBLEM Total processes is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS WARNING: 200 processes [04:34:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 133 processes [04:36:52] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [04:47:33] PROBLEM Total processes is now: CRITICAL on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS CRITICAL: 201 processes [05:04:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 196 processes [05:06:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [05:36:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [05:47:03] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [05:47:33] PROBLEM Total processes is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS WARNING: 200 processes [06:04:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 226 processes [06:06:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [06:07:33] PROBLEM Total processes is now: CRITICAL on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS CRITICAL: 201 processes [06:36:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [06:53:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 243 processes [06:56:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 206 processes [07:01:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 200 processes [07:07:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [07:09:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 193 processes [07:28:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 163 processes [07:32:32] PROBLEM Total processes is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS WARNING: 200 processes [07:37:34] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [08:04:35] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 214 processes [08:07:34] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [08:08:44] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [08:33:44] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 193 processes [08:37:35] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [08:48:44] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 203 processes [08:59:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [09:08:04] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [09:33:44] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes [09:38:26] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [10:09:14] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [10:09:34] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 214 processes [10:18:44] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 208 processes [10:33:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 196 processes [10:38:44] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [10:39:14] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [11:04:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [11:09:14] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [11:33:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 160 processes [11:39:16] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [11:44:34] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [11:49:35] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 187 processes [12:08:45] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [12:11:52] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [12:16:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 201 processes [12:21:45] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 192 processes [12:34:34] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 220 processes [12:41:54] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [13:03:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 190 processes [13:09:35] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [13:12:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [13:42:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [13:49:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 184 processes [13:53:46] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 207 processes [13:58:45] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes [14:12:34] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [14:42:34] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [14:59:35] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 211 processes [15:12:43] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [15:14:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 184 processes [15:42:43] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [16:13:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [16:14:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [16:19:35] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [16:43:24] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [16:58:45] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 193 processes [17:13:25] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [17:38:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 208 processes [17:43:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [17:47:33] RECOVERY Total processes is now: OK on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS OK: 147 processes [17:48:13] RECOVERY Free ram is now: OK on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: OK: 94% free memory [17:58:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 164 processes [18:08:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 208 processes [18:14:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [18:20:32] PROBLEM Total processes is now: CRITICAL on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS CRITICAL: 277 processes [18:40:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 230 processes [18:44:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [18:46:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 358 processes [18:50:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 134 processes [18:54:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 542 processes [19:14:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [19:44:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [19:53:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 254 processes [20:14:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [20:30:12] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 92% free memory [20:45:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [21:13:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 193 processes [21:16:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [21:18:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 238 processes [21:26:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 198 processes [21:28:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 191 processes [21:31:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 201 processes [21:41:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 198 processes [21:47:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [21:59:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [22:04:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 239 processes [22:17:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [22:48:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [23:19:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [23:24:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 198 processes [23:26:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [23:34:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 205 processes [23:43:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 194 processes [23:46:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 205 processes [23:49:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [23:51:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 193 processes [23:54:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 218 processes [23:59:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 100 processes