[00:01:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [00:03:32] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.41, 1.92, 4.64 [00:05:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 1.50, 1.86, 4.50 [00:36:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [00:37:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [00:39:33] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 21% free memory [00:40:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [00:40:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [00:41:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [00:45:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [00:48:13] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [00:48:53] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [00:59:43] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [01:02:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 18% free memory [01:16:23] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 155 processes [01:33:52] PROBLEM Current Load is now: CRITICAL on nova-salt-minion4.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:34:32] PROBLEM Disk Space is now: CRITICAL on nova-salt-minion4.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:35:13] PROBLEM Free ram is now: CRITICAL on nova-salt-minion4.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:36:43] PROBLEM Total processes is now: CRITICAL on nova-salt-minion4.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:37:23] PROBLEM dpkg-check is now: CRITICAL on nova-salt-minion4.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [04:37:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 22% free memory [04:38:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [04:38:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [04:39:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [04:45:33] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 18% free memory [05:01:13] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [05:02:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [05:06:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [06:31:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS WARNING: 151 processes [06:36:53] RECOVERY Total processes is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS OK: 147 processes [06:38:33] PROBLEM dpkg-check is now: CRITICAL on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: DPKG CRITICAL dpkg reports broken packages [06:39:43] PROBLEM dpkg-check is now: CRITICAL on staging.pmtpa.wmflabs 10.4.1.24 output: DPKG CRITICAL dpkg reports broken packages [06:44:32] PROBLEM dpkg-check is now: CRITICAL on build-lucid1.pmtpa.wmflabs 10.4.1.53 output: DPKG CRITICAL dpkg reports broken packages [06:44:47] RECOVERY dpkg-check is now: OK on staging.pmtpa.wmflabs 10.4.1.24 output: All packages OK [06:45:33] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 25% free memory [06:46:13] PROBLEM dpkg-check is now: CRITICAL on ve-change-marking.pmtpa.wmflabs 10.4.0.211 output: DPKG CRITICAL dpkg reports broken packages [06:46:23] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [06:48:33] PROBLEM Disk Space is now: CRITICAL on jesusaurus-cleanup.pmtpa.wmflabs 10.4.0.233 output: DISK CRITICAL - free space: / 0 MB (0% inode=7%): [06:49:33] RECOVERY dpkg-check is now: OK on build-lucid1.pmtpa.wmflabs 10.4.1.53 output: All packages OK [06:51:12] RECOVERY dpkg-check is now: OK on ve-change-marking.pmtpa.wmflabs 10.4.0.211 output: All packages OK [06:51:22] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 22% free memory [07:01:53] PROBLEM dpkg-check is now: CRITICAL on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: DPKG CRITICAL dpkg reports broken packages [07:02:43] PROBLEM Free ram is now: CRITICAL on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Critical: 5% free memory [07:07:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [07:38:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.74, 9.75, 6.30 [07:41:23] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 10.28, 9.32, 6.59 [07:44:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 10.83, 9.50, 6.46 [08:37:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [08:40:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [08:41:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [08:49:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [08:53:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [09:00:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [09:06:32] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.60, 1.91, 4.46 [09:50:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 2.13, 1.92, 4.41 [10:03:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 0.20, 0.82, 3.75 [11:49:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 169 processes [12:19:22] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 150 processes [12:27:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 152 processes [12:39:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [12:40:43] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [12:41:13] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:49:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [12:52:53] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [12:58:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [15:12:23] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [15:37:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [16:12:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 192 processes [16:32:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 208 processes [16:37:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [16:38:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [16:38:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:39:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:56:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [16:57:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:05:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [17:09:52] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 19% free memory [17:11:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [17:12:23] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 195 processes [18:50:23] PROBLEM dpkg-check is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: DPKG CRITICAL dpkg reports broken packages [19:00:22] RECOVERY dpkg-check is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: All packages OK [19:47:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 270 processes [20:18:52] PROBLEM Current Load is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [20:19:32] PROBLEM Disk Space is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [20:20:13] PROBLEM Free ram is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [20:21:43] PROBLEM Total processes is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [20:23:53] RECOVERY Current Load is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: OK - load average: 0.33, 0.81, 0.51 [20:24:33] RECOVERY Disk Space is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: DISK OK [20:25:12] RECOVERY Free ram is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: OK: 97% free memory [20:26:42] RECOVERY Total processes is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 120 processes [20:37:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:40:53] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [20:41:43] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [20:53:53] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [20:55:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [20:58:52] PROBLEM Current Load is now: CRITICAL on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [20:59:32] PROBLEM Disk Space is now: CRITICAL on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [20:59:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [21:00:12] PROBLEM Free ram is now: CRITICAL on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [21:01:42] PROBLEM Total processes is now: CRITICAL on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [21:02:22] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: DPKG CRITICAL dpkg reports broken packages [21:03:52] RECOVERY Current Load is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: OK - load average: 0.58, 0.84, 0.52 [21:04:32] RECOVERY Disk Space is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: DISK OK [21:05:12] RECOVERY Free ram is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: OK: 89% free memory [21:06:42] RECOVERY Total processes is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: PROCS OK: 84 processes [21:07:22] RECOVERY dpkg-check is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.1.85 output: All packages OK [22:19:52] PROBLEM Free ram is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Critical: 5% free memory [22:23:53] PROBLEM Current Load is now: CRITICAL on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [22:24:33] PROBLEM Disk Space is now: CRITICAL on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [22:25:12] PROBLEM Free ram is now: CRITICAL on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [22:26:42] PROBLEM Total processes is now: CRITICAL on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [22:27:22] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [22:28:52] RECOVERY Current Load is now: OK on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: OK - load average: 0.73, 0.93, 0.54 [22:29:32] RECOVERY Disk Space is now: OK on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: DISK OK [22:30:12] RECOVERY Free ram is now: OK on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: OK: 89% free memory [22:31:42] RECOVERY Total processes is now: OK on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: PROCS OK: 91 processes [22:32:22] RECOVERY dpkg-check is now: OK on mwreview-dev6.pmtpa.wmflabs 10.4.1.90 output: All packages OK [22:39:52] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 64% free memory [22:48:52] PROBLEM Current Load is now: CRITICAL on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: Connection refused by host [22:49:32] PROBLEM Disk Space is now: CRITICAL on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: Connection refused by host [22:50:12] PROBLEM Free ram is now: CRITICAL on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: Connection refused by host [22:51:42] PROBLEM Total processes is now: CRITICAL on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: Connection refused by host [22:52:22] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: Connection refused by host [22:53:52] RECOVERY Current Load is now: OK on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: OK - load average: 0.92, 1.04, 0.59 [22:54:32] RECOVERY Disk Space is now: OK on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: DISK OK [22:55:12] RECOVERY Free ram is now: OK on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: OK: 89% free memory [22:56:42] RECOVERY Total processes is now: OK on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: PROCS OK: 84 processes [22:57:22] RECOVERY dpkg-check is now: OK on mwreview-dev7.pmtpa.wmflabs 10.4.1.91 output: All packages OK