[00:05:12] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 17% free memory [00:05:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 192 processes [00:15:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 22% free memory [00:15:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 150 processes [00:18:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:31:53] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 91% free memory [00:33:23] RECOVERY Free ram is now: OK on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: OK: 92% free memory [00:49:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:19:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:26:42] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [01:31:42] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [01:36:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 198 processes [01:51:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:21:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:51:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:21:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:41:42] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 195 processes [03:54:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:23:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 203 processes [04:24:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:28:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 118 processes [04:56:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:26:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:42:23] RECOVERY dpkg-check is now: OK on math-test.pmtpa.wmflabs 10.4.1.107 output: All packages OK [05:45:23] PROBLEM Free ram is now: UNKNOWN on math-test.pmtpa.wmflabs 10.4.1.107 output: NRPE: Unable to read output [05:45:53] RECOVERY Current Load is now: OK on math-test.pmtpa.wmflabs 10.4.1.107 output: OK - load average: 0.09, 0.29, 0.15 [05:46:43] RECOVERY Total processes is now: OK on math-test.pmtpa.wmflabs 10.4.1.107 output: PROCS OK: 92 processes [05:49:33] RECOVERY Disk Space is now: OK on math-test.pmtpa.wmflabs 10.4.1.107 output: DISK OK [05:56:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:29:54] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:59:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 18% free memory [07:01:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:19:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [07:27:22] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 18% free memory [07:31:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:01:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:22:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [08:34:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:04:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:30:33] PROBLEM Free ram is now: WARNING on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Warning: 19% free memory [09:34:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:04:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:06:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:05:42] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [13:06:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:26:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 19% free memory [13:31:22] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 28% free memory [13:39:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:09:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:39:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:50:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 9% free memory [14:50:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 225 processes [15:00:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 199 processes [15:05:22] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 26% free memory [15:06:02] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 105 processes [15:10:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:41:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:08:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 15% free memory [16:11:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:28:53] PROBLEM Current Load is now: CRITICAL on openstack-role-dev8.pmtpa.wmflabs 10.4.1.113 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:29:33] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev8.pmtpa.wmflabs 10.4.1.113 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:30:13] PROBLEM Free ram is now: CRITICAL on openstack-role-dev8.pmtpa.wmflabs 10.4.1.113 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:31:43] PROBLEM Total processes is now: CRITICAL on openstack-role-dev8.pmtpa.wmflabs 10.4.1.113 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:32:23] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev8.pmtpa.wmflabs 10.4.1.113 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:40:43] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 22% free memory [16:41:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:08:43] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [17:11:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:20:53] PROBLEM Current Load is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.114 output: Connection refused by host [17:21:33] PROBLEM Disk Space is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.114 output: Connection refused by host [17:21:43] PROBLEM Total processes is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.114 output: Connection refused by host [17:22:13] PROBLEM Free ram is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.114 output: Connection refused by host [17:22:23] PROBLEM dpkg-check is now: CRITICAL on mwreview-test.pmtpa.wmflabs 10.4.1.114 output: Connection refused by host [17:41:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:53:53] PROBLEM Current Load is now: CRITICAL on deployment-el-test.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [17:54:33] PROBLEM Disk Space is now: CRITICAL on deployment-el-test.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:55:13] PROBLEM Free ram is now: CRITICAL on deployment-el-test.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:56:43] PROBLEM Total processes is now: CRITICAL on deployment-el-test.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:57:23] PROBLEM dpkg-check is now: CRITICAL on deployment-el-test.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:14:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:23:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 160 processes [18:27:23] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 10.37, 9.24, 6.67 [18:31:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 8.76, 8.22, 6.37 [18:32:12] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 7.42, 7.44, 5.90 [18:32:42] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 7.47, 7.32, 5.93 [18:36:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 6.91, 6.40, 5.47 [18:41:53] PROBLEM Total processes is now: WARNING on tools-login.pmtpa.wmflabs 10.4.0.220 output: PROCS WARNING: 157 processes [18:44:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:01:33] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 0.26, 2.51, 4.47 [19:02:13] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 0.12, 2.48, 4.84 [19:02:43] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 0.08, 2.29, 4.79 [19:06:33] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 0.16, 1.25, 4.03 [19:07:23] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 0.07, 1.07, 4.17 [19:15:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:46:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:16:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:38:43] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 22% free memory [20:46:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:16:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:26:43] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [21:31:43] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 20% free memory [21:46:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:17:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:26:53] RECOVERY Total processes is now: OK on tools-login.pmtpa.wmflabs 10.4.0.220 output: PROCS OK: 150 processes [22:48:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:19:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:49:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3)