[09:13:38] Channel is now logged [09:48:53] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 19% free memory [12:37:43] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 27% free memory [12:38:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [12:38:53] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 21% free memory [12:40:33] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 23% free memory [12:41:13] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:41:53] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [12:48:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 18% free memory [12:51:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [12:51:52] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 19% free memory [13:04:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [13:04:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [13:05:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [13:11:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 160 processes [13:51:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [13:54:52] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-upload-test5.pmtpa.wmflabs 10.4.1.86 output: DPKG CRITICAL dpkg reports broken packages [13:56:13] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 151 processes [13:59:53] RECOVERY dpkg-check is now: OK on deployment-cache-upload-test5.pmtpa.wmflabs 10.4.1.86 output: All packages OK [14:01:12] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 149 processes [14:09:13] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 159 processes [14:31:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 198 processes [14:44:32] PROBLEM Disk Space is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [14:45:12] PROBLEM Free ram is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [14:45:52] PROBLEM Current Load is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [14:46:42] PROBLEM Total processes is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [14:47:22] PROBLEM dpkg-check is now: CRITICAL on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [14:49:32] RECOVERY Disk Space is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: DISK OK [14:50:13] RECOVERY Free ram is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: OK: 97% free memory [14:50:53] RECOVERY Current Load is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: OK - load average: 0.31, 0.72, 0.47 [14:51:43] RECOVERY Total processes is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 122 processes [15:12:22] RECOVERY dpkg-check is now: OK on gerrit-tomcat.pmtpa.wmflabs 10.4.0.81 output: All packages OK [15:21:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [16:06:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [16:11:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [16:29:13] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 147 processes [16:38:33] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 23% free memory [16:39:13] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:39:53] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [16:40:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [16:41:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:46:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 18% free memory [16:48:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 19% free memory [16:57:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:07:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [17:09:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [17:16:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 195 processes [17:35:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 5.86, 6.42, 5.48 [17:46:58] PROBLEM Free ram is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Critical: 5% free memory [18:00:32] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.62, 2.92, 4.49 [19:11:53] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 6% free memory [19:21:53] PROBLEM Free ram is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Critical: 5% free memory [19:36:52] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 6% free memory [19:51:24] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 197 processes [19:53:54] PROBLEM Current Load is now: CRITICAL on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [19:54:34] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [19:55:13] PROBLEM Free ram is now: CRITICAL on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [19:56:43] PROBLEM Total processes is now: CRITICAL on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [19:57:23] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [19:58:53] RECOVERY Current Load is now: OK on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: OK - load average: 0.91, 0.93, 0.53 [19:59:33] RECOVERY Disk Space is now: OK on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: DISK OK [20:00:22] RECOVERY Free ram is now: OK on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: OK: 85% free memory [20:01:42] RECOVERY Total processes is now: OK on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: PROCS OK: 92 processes [20:02:22] RECOVERY dpkg-check is now: OK on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: All packages OK [20:17:43] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: DISK CRITICAL - free space: / 286 MB (2% inode=70%): [20:26:53] PROBLEM Free ram is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Critical: 4% free memory [20:37:13] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:37:53] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [20:38:43] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 27% free memory [20:39:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [20:41:33] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 22% free memory [20:56:43] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [20:56:53] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 6% free memory [21:00:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [21:00:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [21:04:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 18% free memory [21:07:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [21:52:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 1.67, 4.75, 5.07 [21:53:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 3.99, 5.45, 5.11 [21:57:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.48, 3.55, 4.42 [22:01:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 10.09, 8.92, 6.13 [22:05:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 9.98, 7.98, 6.10 [23:56:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [23:57:02] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 2.91, 2.99, 4.64