[01:15:53] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 205 processes [01:20:52] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 96 processes [01:23:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 140 processes [04:07:12] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 14% free memory [04:12:13] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 4% free memory [04:21:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 172 processes [04:22:12] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 72% free memory [04:31:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 148 processes [07:05:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 17% free memory [07:10:53] PROBLEM Current Load is now: CRITICAL on moodle-24.pmtpa.wmflabs 10.4.0.92 output: Connection refused by host [07:11:33] PROBLEM Disk Space is now: CRITICAL on moodle-24.pmtpa.wmflabs 10.4.0.92 output: Connection refused by host [07:11:43] PROBLEM Total processes is now: CRITICAL on moodle-24.pmtpa.wmflabs 10.4.0.92 output: Connection refused by host [07:12:13] PROBLEM Free ram is now: CRITICAL on moodle-24.pmtpa.wmflabs 10.4.0.92 output: Connection refused by host [07:12:23] PROBLEM dpkg-check is now: CRITICAL on moodle-24.pmtpa.wmflabs 10.4.0.92 output: Connection refused by host [07:20:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 21% free memory [08:23:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:27:12] PROBLEM Current Load is now: WARNING on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: WARNING - load average: 16.89, 13.12, 7.47 [08:27:52] PROBLEM Total processes is now: WARNING on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: PROCS WARNING: 163 processes [08:32:23] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [08:42:53] RECOVERY Total processes is now: OK on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: PROCS OK: 106 processes [08:54:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:57:13] RECOVERY Current Load is now: OK on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: OK - load average: 0.09, 0.62, 4.88 [09:24:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:54:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:24:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:32:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 187 processes [10:37:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 149 processes [10:54:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:55:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 189 processes [11:24:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:56:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:26:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:56:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:26:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:56:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:26:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:56:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:28:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:59:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:29:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:59:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:29:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:59:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:29:44] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:36:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [18:51:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [19:00:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:31:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:01:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:31:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:36:13] PROBLEM Total processes is now: CRITICAL on bots-apache01.pmtpa.wmflabs 10.4.0.141 output: PROCS CRITICAL: 242 processes [21:01:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:31:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:59:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 153 processes [22:01:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:31:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:01:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:04:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 142 processes [23:21:12] RECOVERY Total processes is now: OK on bots-apache01.pmtpa.wmflabs 10.4.0.141 output: PROCS OK: 116 processes [23:31:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3)