[00:39:23] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 25% free memory [00:41:43] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [00:52:22] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 17% free memory [01:04:42] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [01:06:42] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 175 processes [01:16:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 96 processes [01:30:43] PROBLEM Free ram is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:35:42] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [01:48:42] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 9% free memory [04:37:22] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 25% free memory [04:39:42] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [04:41:24] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [04:47:43] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [04:54:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [05:04:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [05:10:23] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 17% free memory [05:48:33] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 18% free memory [06:30:23] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 33% free memory [06:32:33] PROBLEM Total processes is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS WARNING: 155 processes [06:33:33] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 27% free memory [06:33:43] PROBLEM Free ram is now: CRITICAL on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Critical: 5% free memory [06:38:42] PROBLEM Free ram is now: WARNING on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Warning: 6% free memory [06:48:02] PROBLEM Free ram is now: CRITICAL on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:48:52] PROBLEM Free ram is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:44] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 16% free memory [06:56:27] Damianz: bots-bnr1 [06:58:13] we'll need a package for tcl8.6-tip-386-impl, i guess [06:58:43] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 57% free memory [07:02:33] RECOVERY Total processes is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS OK: 150 processes [07:41:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.87, 5.79, 5.37 [08:37:53] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [08:39:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [08:46:44] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.02, 4.20, 4.81 [08:50:52] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [08:57:33] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [09:19:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 73% free memory [09:20:13] PROBLEM dpkg-check is now: CRITICAL on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: DPKG CRITICAL dpkg reports broken packages [09:24:14] !log bots root: sql2 upgraded system [09:24:17] Logged the message, Master [09:24:41] !log bots root: /data/project/testfile is -rw-r--r-- 1 root root 619511808 2013-01-24 22:24 testfile - wtf?? [09:24:42] Logged the message, Master [09:25:12] RECOVERY dpkg-check is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: All packages OK [09:28:30] !log bots root: rm /data/project/testfile [09:28:32] Logged the message, Master [09:29:57] Damianz who is creating /data/project/mysql/ [09:32:23] petan: did you get my message on your enwp talk page? [09:32:42] maybe... [09:32:45] let me check [09:33:00] aha [09:33:47] !log bots added rschen7754 [09:33:48] Logged the message, Master [09:33:54] thx [09:34:01] !botsdocs [09:34:01] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [09:37:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [09:39:27] !log bots root: maintenance on bots-sql2 in 20 minutes [09:39:29] Logged the message, Master [10:02:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 75% free memory [11:19:27] :-( [11:19:39] bots-sql2 still down? [11:35:40] Beetstra yes [11:35:57] not even in 50% [11:35:59] that's why my bot keeps complaining .. :-D [11:36:16] -> Go away, Wikimedia/Beetstra, you can't do that. [11:57:58] haha [12:01:53] fuck I hate to forget to commit [12:02:03] when I am in office at friday [12:02:34] whole weekend having old code [12:11:36] petan happens to the best of us [12:30:14] !log bots petrb: sql2 is running [12:30:17] Logged the message, Master [12:31:52] Beetstra up [12:37:33] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [12:40:52] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [12:58:07] petan: crond [12:58:17] It contains backups of the sql databases [12:58:53] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [13:05:32] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [13:15:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 19% free memory [13:19:32] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [13:21:59] Damianz all of them? [13:22:12] does it run on all severs? how u merge them? [14:49:33] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:51:54] PROBLEM Current Load is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:52:34] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:52:45] PROBLEM SSH is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Server answer: [14:52:54] PROBLEM Disk Space is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:01:54] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 0.66, 1.16, 1.07 [15:02:34] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [15:02:44] RECOVERY SSH is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [15:02:54] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK [15:04:24] RECOVERY Total processes is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS OK: 142 processes [15:04:44] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 35% free memory [15:09:32] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 30% free memory [15:11:33] PROBLEM Total processes is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS WARNING: 151 processes [16:38:53] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [16:40:32] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [16:43:01] Hi, pls. can someone help me https://bugzilla.wikimedia.org/show_bug.cgi?id=44398 [16:48:55] Krenair: can you assist me, pls. https://bugzilla.wikimedia.org/show_bug.cgi?id=44398 [16:49:18] I want to use "git review ..." on my instance. [16:49:25] but there's a problem with that [16:49:44] see traceback in the bugzilla [17:01:52] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 17% free memory [17:08:32] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:32:45] Wikinaut, you know you could just follow the instructions it gives you [17:33:49] Or actually maybe that wouldn't work.. hm.. [17:34:00] honestly, I do not know what is wrong, otherwise I would have ask. [17:34:18] Can you "git review" from your instance/s ? [17:34:24] then send me your config [17:34:37] I think, it's a https / ssh problem .... [17:34:52] may be a general problem [17:35:11] I only have access to the openstack dev project, and I don't really want to install anything like git-review on there... [17:40:53] Wikinaut, When I do 'ssh -p 29418 gerrit.wikimedia.org' I get a publickey error... [17:41:20] What about you? [17:42:45] Wikinaut, ah... Everything is fine if I run my ssh commands for bastion and the instance with -A (forward authentication) [17:47:59] Wikinaut, hello? [17:48:10] ahem, here I am [17:49:01] "Permission denied (publickey)." [17:49:23] So exit back to bastion and ssh into your instance with -A this time [17:49:35] Now try ssh -p 29418 gerrit.wikimedia.org [17:51:33] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory [17:52:07] Wikinaut? [17:53:17] Krenair: need to modify my PuTTY . one moment [17:56:12] Krenai: cannot test this at the moment. Need to look up for description of plink.exe, a local proxy which makes the bastion-tunnel/proxy connection [17:56:55] "plink.exe bastion.wmflabs.org -l wikinaut -agent -nc %host:%port" [17:57:25] this is what PuTTY uses currently see https://labsconsole.wikimedia.org/wiki/User:Wikinaut/Help:Access_to_instances_with_PuTTY_and_WinSCP#How_to_set_up_PuTTY_for_proxying_through_bastion.wmflabs.org_to_your_instance [17:57:53] let's close this issue for the moment [17:58:18] I am busy with new version of E:OpenID which will be published this evening [17:58:31] okay ? [18:02:21] Wikinaut, so you have auth set up to connect through to your instance already [18:02:45] When you're in bastion you just need to go to your instance with the -A argument [18:05:47] Wikinaut, ah, okay so I've read that page... I think you want 'Allow agent forwarding' in Connection -> SSH -> Auth -> 'Authentication parameters' [18:06:14] As said above, I need to check with this plink.exe what the equivalent option for the -A argument is [18:06:26] ^yes [18:07:16] just to make clear: what I documented on the wiki page works perfectly, except... [18:07:23] ... this git review stuff [18:07:36] would be nice, if we can solve this, too [18:07:48] (but perhaps, it's not possible) [18:10:45] Wikinaut, I'm pretty sure this is an SSH key problem, nothing to do with git-review. [18:10:56] hmm [18:11:11] will double-check this (later) [18:12:54] Krenair: look, logging in (Pageant has the key) through bastion to my instance works. [18:13:11] Right, but you need to forward the agent to the instance. [18:13:23] The key for instance isthe same key as for gerrit, isn't it [18:13:25] ? [18:13:29] ^yes [18:13:45] so I'm logged in in the instance with the gerrit key [18:13:55] Not quite sure. [18:13:58] but cannot login to gerrit with it [18:14:03] Right [18:14:19] To solve this, you very simply add -A to the command which you use when going from bastion to the instance [18:14:29] so, another step of "forwarding" is somehow broken [18:14:36] Krenair: ok [18:14:41] will try it (later) [18:14:41] By default, this is not on [18:14:46] You need to explicitly enable it [18:15:09] the point is, that I need to understand this plink.exe [18:15:15] (part of the PuTTY suite) [18:15:27] perhaps there is an additional parameter [18:15:32] Why are you even using PuTTY anyway? Trying to connect from a windows machine or something? [18:15:38] yes [18:15:43] only as terminal [18:15:45] Ugh. [18:15:49] argh [18:15:53] I hate windows [18:16:14] something different [18:16:42] ^delete this [18:17:48] http://pastebin.com/UhEkhA6h [18:29:56] Wikinaut delete what [18:30:04] from logs? [18:30:22] "something different" [18:30:34] ;-) [18:30:36] ignore it [18:30:40] lol ok [18:31:37] Krenair: was the pastebin for me ? hash is invalid [18:31:56] It had a 10 minute expiry [18:32:21] http://pastebin.com/A9WFDh7e [18:38:05] ty [18:41:33] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 28% free memory [19:11:03] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [19:16:02] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 8% free memory [19:19:12] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 152 processes [19:29:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 6.97, 6.66, 5.72 [20:23:33] Damianz: are you still thinking of installing the software i need on bnr1? [20:38:32] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:41:53] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 25% free memory [21:04:53] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 17% free memory [21:06:32] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [22:59:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.90, 4.91, 4.99 [23:07:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.84, 4.99, 5.02 [23:17:27] Ryan_Lane: +++ good news +++ I managed to solve all E:OpenID problems. Currently, I am heavily testing before I commit this version to gerrit [23:22:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 5.27, 4.98, 4.99 [23:52:42] Ryan_Lane: [23:52:44] ping