[01:09:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 210 processes [01:19:42] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 94 processes [01:21:47] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 3.93, 6.31, 5.48 [01:26:43] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.49, 4.54, 4.94 [02:09:03] PROBLEM Disk Space is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK WARNING - free space: / 447 MB (4% inode=72%): [06:34:03] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [06:59:02] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [08:54:03] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [09:27:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [11:02:03] PROBLEM Disk Space is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK WARNING - free space: / 573 MB (5% inode=72%): [11:09:23] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [11:27:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [11:55:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [11:59:43] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 10.10, 8.30, 6.17 [12:21:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 148 processes [12:39:42] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 226 processes [12:39:52] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 153 processes [12:44:43] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 4.76, 4.15, 4.87 [12:47:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 292 processes [12:49:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 257 processes [12:50:53] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 319 processes [12:52:43] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 4.44, 4.92, 5.08 [13:55:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [14:03:53] PROBLEM Total processes is now: WARNING on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: PROCS WARNING: 182 processes [14:04:13] PROBLEM Current Load is now: WARNING on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: WARNING - load average: 19.11, 14.37, 8.80 [14:08:52] RECOVERY Total processes is now: OK on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: PROCS OK: 101 processes [14:19:12] RECOVERY Current Load is now: OK on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: OK - load average: 0.03, 1.45, 4.80 [14:28:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [16:28:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [16:55:53] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 154 processes [16:56:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [16:57:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 198 processes [16:59:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 165 processes [17:00:52] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 142 processes [17:07:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 273 processes [17:09:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 295 processes [17:13:53] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 260 processes [17:40:43] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [18:02:03] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [18:56:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [19:24:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [21:24:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [21:57:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [22:50:13] PROBLEM Disk Space is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK WARNING - free space: / 573 MB (5% inode=72%): [23:57:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory