[00:01:46] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 4.06, 3.97, 4.80 [00:03:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [00:06:44] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [00:21:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [00:37:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [00:40:54] RECOVERY Free ram is now: OK on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: OK: 33% free memory [00:41:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [00:50:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [00:52:53] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [01:01:32] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 17% free memory [01:07:44] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 175 processes [01:10:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 15% free memory [01:12:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 100 processes [01:23:58] PROBLEM Free ram is now: WARNING on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: Warning: 19% free memory [01:34:56] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [02:08:49] !wikibugs [02:08:52] !bots [02:08:53] http://www.mediawiki.org/wiki/Wikimedia_Labs/Create_a_bot_running_infrastructure proposal for bots [02:29:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [02:37:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [02:48:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 6.81, 6.60, 5.72 [03:13:54] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.83, 4.21, 4.76 [04:34:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 6.34, 6.29, 5.36 [04:37:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [04:38:55] RECOVERY Free ram is now: OK on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: OK: 33% free memory [04:40:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 25% free memory [04:41:32] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 20% free memory [04:41:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 4.96, 5.44, 5.25 [04:50:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [04:53:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [05:00:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [05:06:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 5.69, 6.09, 5.28 [05:09:33] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 18% free memory [05:09:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.78, 4.29, 4.91 [05:11:54] PROBLEM Free ram is now: WARNING on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: Warning: 19% free memory [05:21:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.92, 4.53, 4.98 [05:36:42] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 3.28, 4.22, 4.80 [05:58:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [06:31:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:31:54] RECOVERY Free ram is now: OK on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: OK: 52% free memory [06:36:54] PROBLEM dpkg-check is now: CRITICAL on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: DPKG CRITICAL dpkg reports broken packages [06:36:55] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 147 processes [06:39:24] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [06:41:55] RECOVERY dpkg-check is now: OK on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: All packages OK [07:11:22] PROBLEM dpkg-check is now: CRITICAL on bots-4.pmtpa.wmflabs 10.4.0.64 output: DPKG CRITICAL dpkg reports broken packages [07:14:32] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 20% free memory [07:33:16] !log bots petrb: public_html is failing, changing to a+rx in a loop [07:33:18] Logged the message, Master [07:33:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [07:56:58] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [07:59:20] test [07:59:25] !ping [07:59:26] pong [08:01:18] Damianz, jeremyb ping [08:01:24] please read !mail [08:01:35] there is urgent issue on apaches [08:05:34] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory [08:10:47] !ping [08:10:47] pong [08:11:27] !ping [08:11:27] pong [08:13:42] petan: Does the user exist? [08:13:49] which user? [08:13:59] read this: https://bugzilla.wikimedia.org/show_bug.cgi?id=43896 [08:14:00] the one the dir gets chmod to 000 for [08:14:11] it's root root [08:14:19] The user has to exist and be part of the bots project [08:14:24] it's not a user [08:14:28] it's whole folder [08:14:37] . /data/project/public_html [08:17:24] because there's a symlink [08:17:38] /data/project/public_html/public_html -> /data/project/public_html [08:17:43] public_html isn't a valid user [08:17:47] so it gets disabled [08:17:57] and because it's a symlink back to root, it does thje whole folder [08:18:11] why is it there? [08:18:16] dunno [08:18:18] just gonna remove it [08:18:22] ok [08:18:38] and why does it chmod 000 non existing users? [08:18:42] what is purpose [08:19:29] If we suspend a user they get removed from projects, so we should disable anything public - the simplest way is to chmod their folder to 000, which means if we need to re-enable it we can just chmod it back and everything will work again [08:22:31] Damianz can we make that check ignore symlinks? [08:22:44] yes [08:22:54] I'll do it when I get in from work today [08:23:30] k [08:38:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [08:40:33] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 23% free memory [08:41:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [09:06:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [09:08:34] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory [09:14:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [09:59:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [10:32:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [12:02:45] !q1 [12:02:45] Damianz where is teh ramcheck in puppet? [12:37:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [12:38:34] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 22% free memory [13:01:33] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory [13:10:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [13:19:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.71, 4.79, 5.00 [13:20:15] !q1 [13:20:16] Damianz where is teh ramcheck in puppet? [13:21:37] tunnel [13:21:46] !tunnel [13:21:46] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [13:21:47] hah [13:44:24] PROBLEM Free ram is now: UNKNOWN on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: NRPE: Call to fork() failed [13:48:25] PROBLEM Total processes is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:48:45] PROBLEM Current Users is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:48:55] PROBLEM Current Load is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:49:24] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:49:34] PROBLEM dpkg-check is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:49:44] PROBLEM Disk Space is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:50:32] PROBLEM SSH is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Server answer: [13:53:22] RECOVERY Total processes is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS OK: 139 processes [13:53:42] RECOVERY Current Users is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: USERS OK - 0 users currently logged in [13:53:52] RECOVERY Current Load is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK - load average: 1.40, 1.78, 1.30 [13:54:23] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 55% free memory [13:54:32] RECOVERY dpkg-check is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: All packages OK [13:54:42] RECOVERY Disk Space is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: DISK OK [13:55:33] RECOVERY SSH is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [15:00:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [15:13:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [16:38:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [16:41:27] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [16:51:45] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 20% free memory [16:59:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [17:03:59] http://www.quinnnorton.com/said/?p=641 http://lists.freeculture.org/pipermail/discuss/2013-January/007109.html [17:04:04] errr, whoops [17:04:18] that was supposed to be somewhere else. but i guess feel free to click. ;-( [17:11:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [17:31:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [17:39:32] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory [18:14:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [18:19:46] Logging traf [18:21:45] Logging stopped [18:34:43] !ping [18:34:44] pong [18:34:52] @infobot-detail ping [18:34:52] Info for ping: this key was created at N/A by N/A, this key was displayed 92 time(s), last time at 1/12/2013 6:34:44 PM (00:00:08.4047350 ago) this key is normal [20:04:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [20:12:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [20:37:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [20:39:33] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 21% free memory [20:52:33] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 18% free memory [21:25:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [22:06:44] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [22:30:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [23:03:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory