[02:29:21] PROBLEM SSH is now: CRITICAL on mobile-testing i-00000271 output: CRITICAL - Socket timeout after 10 seconds [02:31:31] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:34:21] RECOVERY SSH is now: OK on mobile-testing i-00000271 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [02:36:31] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 0.29, 0.32, 0.34 [02:46:51] RECOVERY Free ram is now: OK on bots-3 i-000000e5 output: OK: 20% free memory [03:18:10] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:23:00] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in [03:32:50] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:44:50] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [03:48:15] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:58:02] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [03:58:54] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 15% free memory [03:59:54] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [04:05:09] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:09:49] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:13:59] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [04:18:59] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:19:49] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:24:49] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 7% free memory [04:29:49] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [04:29:49] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:34:09] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [04:53:19] PROBLEM Free ram is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:53:19] PROBLEM dpkg-check is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:58:09] RECOVERY Free ram is now: OK on mobile-testing i-00000271 output: OK: 84% free memory [04:58:09] RECOVERY dpkg-check is now: OK on mobile-testing i-00000271 output: All packages OK [05:10:19] PROBLEM SSH is now: CRITICAL on mobile-testing i-00000271 output: CRITICAL - Socket timeout after 10 seconds [05:34:21] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 205 processes [05:39:21] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 198 processes [06:29:22] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 203 processes [06:29:51] PROBLEM Total Processes is now: WARNING on aggregator1 i-0000010c output: PROCS WARNING: 199 processes [06:32:05] PROBLEM Total Processes is now: WARNING on deployment-thumbproxy i-0000026b output: PROCS WARNING: 155 processes [06:40:16] PROBLEM Total Processes is now: CRITICAL on aggregator1 i-0000010c output: PROCS CRITICAL: 201 processes [06:44:31] PROBLEM Free ram is now: CRITICAL on grail i-000002c6 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:13] PROBLEM Total Processes is now: CRITICAL on grail i-000002c6 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:46:03] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 14.71, 10.08, 4.73 [06:47:00] PROBLEM Current Users is now: CRITICAL on grail i-000002c6 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:00] RECOVERY Free ram is now: OK on grail i-000002c6 output: OK: 76% free memory [06:54:00] RECOVERY Total Processes is now: OK on grail i-000002c6 output: PROCS OK: 103 processes [06:54:15] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:58:36] RECOVERY Current Users is now: OK on grail i-000002c6 output: USERS OK - 0 users currently logged in [07:03:39] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 10% free memory [07:04:44] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:04:44] PROBLEM Current Users is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:04:44] PROBLEM Total Processes is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:35] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:12] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [07:06:32] PROBLEM Current Users is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:32] PROBLEM Disk Space is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:33] PROBLEM Total Processes is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:37] PROBLEM Current Users is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:38] PROBLEM Current Load is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:38] PROBLEM Total Processes is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:43] PROBLEM dpkg-check is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:43] PROBLEM dpkg-check is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:43] PROBLEM Total Processes is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:50] PROBLEM dpkg-check is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:07:05] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 7.36, 9.43, 8.11 [07:07:15] PROBLEM Free ram is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:00] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 0.90, 4.50, 5.01 [07:08:10] RECOVERY Current Load is now: OK on precise-test i-00000231 output: OK - load average: 1.84, 3.80, 3.54 [07:08:10] RECOVERY Current Users is now: OK on precise-test i-00000231 output: USERS OK - 0 users currently logged in [07:08:10] RECOVERY Total Processes is now: OK on precise-test i-00000231 output: PROCS OK: 88 processes [07:08:22] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [07:09:20] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:10:50] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 21% free memory [07:10:50] RECOVERY Current Users is now: OK on migration1 i-00000261 output: USERS OK - 0 users currently logged in [07:10:50] RECOVERY Disk Space is now: OK on incubator-bot1 i-00000251 output: DISK OK [07:10:50] RECOVERY Total Processes is now: OK on pybal-precise i-00000289 output: PROCS OK: 82 processes [07:10:55] RECOVERY Current Users is now: OK on incubator-bot1 i-00000251 output: USERS OK - 0 users currently logged in [07:10:55] RECOVERY Current Load is now: OK on incubator-bot1 i-00000251 output: OK - load average: 2.08, 4.41, 4.09 [07:10:55] RECOVERY Total Processes is now: OK on incubator-bot1 i-00000251 output: PROCS OK: 132 processes [07:11:00] RECOVERY dpkg-check is now: OK on pybal-precise i-00000289 output: All packages OK [07:11:00] RECOVERY Total Processes is now: OK on migration1 i-00000261 output: PROCS OK: 85 processes [07:11:05] RECOVERY dpkg-check is now: OK on incubator-bot1 i-00000251 output: All packages OK [07:11:05] RECOVERY dpkg-check is now: OK on migration1 i-00000261 output: All packages OK [07:12:05] RECOVERY Free ram is now: OK on migration1 i-00000261 output: OK: 89% free memory [07:12:57] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.29, 1.86, 3.72 [07:14:07] RECOVERY Current Load is now: OK on maps-tilemill1 i-00000294 output: OK - load average: 0.33, 2.70, 4.20 [07:20:37] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.41, 0.99, 3.49 [07:21:57] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 0.55, 1.47, 4.00 [07:25:37] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.17, 0.60, 2.64 [07:45:39] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 19% free memory [07:51:29] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [08:17:59] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 198 processes [08:21:29] PROBLEM Disk Space is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [08:26:19] RECOVERY Disk Space is now: OK on mobile-testing i-00000271 output: DISK OK [08:28:29] PROBLEM Free ram is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [08:38:30] RECOVERY Free ram is now: OK on mobile-testing i-00000271 output: OK: 84% free memory [11:53:45] PROBLEM Free ram is now: WARNING on ganglia-test2 i-00000250 output: Warning: 19% free memory [12:50:35] RECOVERY Free ram is now: OK on bots-3 i-000000e5 output: OK: 20% free memory [12:58:35] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 19% free memory [14:34:30] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [15:17:31] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [15:22:31] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in [15:53:41] PROBLEM Disk Space is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [15:53:41] PROBLEM Free ram is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [15:58:31] RECOVERY Disk Space is now: OK on mobile-testing i-00000271 output: DISK OK [15:58:31] RECOVERY Free ram is now: OK on mobile-testing i-00000271 output: OK: 84% free memory [16:26:41] PROBLEM Disk Space is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [16:44:01] PROBLEM SSH is now: CRITICAL on mobile-testing i-00000271 output: CRITICAL - Socket timeout after 10 seconds [16:48:51] RECOVERY SSH is now: OK on mobile-testing i-00000271 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [17:20:41] PROBLEM Free ram is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [18:18:51] RECOVERY Free ram is now: OK on ganglia-test2 i-00000250 output: OK: 88% free memory [18:56:41] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [19:01:31] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in [19:36:43] That I guess is the channel for scribunto.wmflabs.org, no? [20:02:41] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [20:30:31] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [21:45:06] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=554709 edit summary: /* Proposals */ [21:48:46] Hello all! [21:52:42] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: Critical: 5% free memory [22:50:42] PROBLEM Free ram is now: CRITICAL on aggregator-test1 i-000002bf output: Critical: 5% free memory [23:30:42] PROBLEM Free ram is now: WARNING on aggregator-test1 i-000002bf output: Warning: 6% free memory [23:42:42] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [23:47:32] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in