[00:37:32] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 22% free memory [00:38:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [00:39:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 24% free memory [00:46:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 17% free memory [00:53:40] Ryan_Lane: Did you see the bug for that? [00:53:54] for the 000 file perms? [00:54:05] mhm [00:54:23] which bug number? [00:54:33] 43896 [00:55:11] I think your response was unrelated [00:55:41] oh [00:55:42] wait [00:55:44] it was defo chmoding it [00:55:46] it's the web server doing this? [00:55:50] yeah [00:55:51] are you fucking kidding me? [00:56:21] *shrug* people shouldn't manually fuck around with automatically managed stuff [00:57:04] heh [00:58:01] dunno why you'd symlink a folder to its root but meh, I'll add some handling for that [01:01:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [01:02:16] I think the split-brain is actually not a major problem [01:02:25] it's almost all on metadata for directories [01:02:45] and I think that's timestamp related [01:03:01] I take that back [01:03:06] I found a bad file [01:04:04] nom [01:05:42] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 178 processes [01:07:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 11% free memory [01:08:10] totally just blaming gluster [01:14:40] well, there are definitely some broken files [01:15:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 101 processes [01:21:56] bah [01:22:00] there's *3* broken files [01:54:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 16% free memory [02:02:32] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 16% free memory [02:31:43] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 12% free memory [03:34:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [03:47:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 16% free memory [04:09:34] PROBLEM Current Users is now: CRITICAL on cephticon2.pmtpa.wmflabs 10.4.0.249 output: Connection refused by host [04:09:35] PROBLEM Current Users is now: CRITICAL on cephticon3.pmtpa.wmflabs 10.4.1.72 output: Connection refused by host [04:10:14] PROBLEM Disk Space is now: CRITICAL on cephticon2.pmtpa.wmflabs 10.4.0.249 output: Connection refused by host [04:10:14] PROBLEM Disk Space is now: CRITICAL on cephticon3.pmtpa.wmflabs 10.4.1.72 output: Connection refused by host [04:10:54] PROBLEM Current Load is now: CRITICAL on cephticon2.pmtpa.wmflabs 10.4.0.249 output: Connection refused by host [04:10:54] PROBLEM Free ram is now: CRITICAL on cephticon2.pmtpa.wmflabs 10.4.0.249 output: Connection refused by host [04:10:55] PROBLEM Current Load is now: CRITICAL on cephticon3.pmtpa.wmflabs 10.4.1.72 output: Connection refused by host [04:11:04] PROBLEM Free ram is now: CRITICAL on cephticon3.pmtpa.wmflabs 10.4.1.72 output: Connection refused by host [04:12:24] PROBLEM Total processes is now: CRITICAL on cephticon2.pmtpa.wmflabs 10.4.0.249 output: Connection refused by host [04:12:25] PROBLEM Total processes is now: CRITICAL on cephticon3.pmtpa.wmflabs 10.4.1.72 output: Connection refused by host [04:14:34] PROBLEM dpkg-check is now: CRITICAL on cephticon2.pmtpa.wmflabs 10.4.0.249 output: Connection refused by host [04:15:53] PROBLEM dpkg-check is now: CRITICAL on cephticon3.pmtpa.wmflabs 10.4.1.72 output: Connection refused by host [04:20:13] RECOVERY Disk Space is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: DISK OK [04:20:13] RECOVERY Disk Space is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: DISK OK [04:20:53] RECOVERY dpkg-check is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: All packages OK [04:20:53] RECOVERY Current Load is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: OK - load average: 0.59, 1.11, 0.81 [04:20:54] RECOVERY Free ram is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: OK: 896% free memory [04:20:54] RECOVERY Current Load is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: OK - load average: 0.62, 1.07, 0.76 [04:21:03] RECOVERY Free ram is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: OK: 1632% free memory [04:22:23] RECOVERY Total processes is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: PROCS OK: 84 processes [04:22:24] RECOVERY Total processes is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: PROCS OK: 90 processes [04:24:33] RECOVERY Current Users is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: USERS OK - 0 users currently logged in [04:24:33] RECOVERY Current Users is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: USERS OK - 0 users currently logged in [04:24:34] RECOVERY dpkg-check is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: All packages OK [04:37:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [04:37:32] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 20% free memory [04:45:33] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 17% free memory [05:00:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [06:02:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [06:10:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 16% free memory [06:29:56] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:34:53] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [06:52:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.20, 5.50, 5.23 [07:37:44] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.84, 4.87, 4.98 [08:40:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [08:40:32] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 20% free memory [08:40:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 23% free memory [08:53:32] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 17% free memory [08:58:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 16% free memory [09:03:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [09:10:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.00, 5.07, 5.02 [09:15:44] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.85, 4.94, 4.96 [09:38:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.75, 5.39, 5.17 [11:18:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 5.01, 4.93, 4.99 [11:26:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.09, 5.12, 5.06 [12:01:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.92, 4.87, 4.95 [12:38:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [12:38:33] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 21% free memory [12:44:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 8.72, 7.92, 6.61 [12:51:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [12:56:32] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 18% free memory [13:16:42] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [13:49:44] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.01, 4.13, 4.87 [16:38:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory [16:41:24] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 24% free memory [16:46:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 15% free memory [16:51:01] * Damianz kicks gerrit for making life hard [17:09:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [18:11:41] is Password Reset expected to work in labsconsole? [19:11:55] could someone add me to bastion.wmflabs.org please? (username: Vogone) [20:00:42] !tunnel [20:00:42] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [20:39:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [20:41:33] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 22% free memory [20:52:25] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [21:21:03] PROBLEM dpkg-check is now: CRITICAL on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: Connection refused by host [21:21:23] PROBLEM Current Users is now: CRITICAL on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: Connection refused by host [21:21:53] PROBLEM Free ram is now: CRITICAL on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: Connection refused by host [21:22:13] PROBLEM Total processes is now: CRITICAL on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: Connection refused by host [21:22:43] PROBLEM Current Load is now: CRITICAL on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: Connection refused by host [21:23:13] PROBLEM Disk Space is now: CRITICAL on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: Connection refused by host [21:31:22] RECOVERY Current Users is now: OK on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: USERS OK - 0 users currently logged in [21:31:52] RECOVERY Free ram is now: OK on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: OK: 898% free memory [21:32:12] RECOVERY Total processes is now: OK on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: PROCS OK: 84 processes [21:32:42] RECOVERY Current Load is now: OK on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: OK - load average: 0.35, 1.01, 0.73 [21:34:32] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 18% free memory [21:36:04] RECOVERY dpkg-check is now: OK on pdbhandler-2.pmtpa.wmflabs 10.4.1.73 output: All packages OK [21:39:34] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 29% free memory [21:41:44] PROBLEM Free ram is now: UNKNOWN on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: NRPE: Call to fork() failed [21:45:22] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:45:32] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:46:42] PROBLEM SSH is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Server answer: [21:46:43] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:47:32] PROBLEM Current Users is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:47:42] PROBLEM Current Load is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:47:53] PROBLEM Disk Space is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:50:24] RECOVERY Total processes is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS OK: 128 processes [21:50:34] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [21:51:44] RECOVERY SSH is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [21:51:44] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 53% free memory [21:52:34] RECOVERY Current Users is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: USERS OK - 0 users currently logged in [21:52:44] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 0.15, 0.70, 0.89 [21:52:54] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK