[02:50:10] RECOVERY Current Users is now: OK on utils-abogott utils-abogott output: USERS OK - 0 users currently logged in [02:50:50] RECOVERY Disk Space is now: OK on utils-abogott utils-abogott output: DISK OK [02:51:50] RECOVERY Free ram is now: OK on utils-abogott utils-abogott output: OK: 89% free memory [02:52:50] RECOVERY Total Processes is now: OK on utils-abogott utils-abogott output: PROCS OK: 76 processes [02:54:50] RECOVERY Current Load is now: OK on utils-abogott utils-abogott output: OK - load average: 0.01, 0.11, 0.11 [06:05:00] PROBLEM Free ram is now: WARNING on bots-2 bots-2 output: Warning: 18% free memory [06:45:00] RECOVERY Free ram is now: OK on bots-2 bots-2 output: OK: 23% free memory [08:49:30] PROBLEM Free ram is now: WARNING on bots-2 bots-2 output: Warning: 18% free memory [09:54:30] RECOVERY Free ram is now: OK on bots-2 bots-2 output: OK: 20% free memory [12:19:30] PROBLEM Free ram is now: WARNING on bots-2 bots-2 output: Warning: 19% free memory [12:55:40] PROBLEM Free ram is now: CRITICAL on bots-3 bots-3 output: Critical: 5% free memory [13:14:28] Hey, is bots-apache1 playing up? [13:17:06] no? [14:30:40] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 6% free memory [14:41:32] New patchset: Mark Bergsma; "Merge commit 'origin/production~14^2' into test" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2294 [14:42:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2294 [14:42:16] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2294 [14:45:40] PROBLEM Free ram is now: CRITICAL on bots-3 bots-3 output: Critical: 5% free memory [14:50:40] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 7% free memory [15:27:30] New patchset: Mark Bergsma; "Revert "Merge remote-tracking branch 'origin/production' into test"" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2298 [15:27:56] New patchset: Mark Bergsma; "Changed xmit_hash_policy to a variable so that we can set it to any policy desired" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2299 [15:28:16] New patchset: Mark Bergsma; "upgrading dbs 13,18,25,33" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2300 [15:28:35] New patchset: Mark Bergsma; "add nagios to iptables, enable nagios to talk to swift via nrpe" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2301 [15:28:55] New patchset: Mark Bergsma; "changed URL for nagios check to ms-fe to something that exists" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2302 [15:29:15] New patchset: Mark Bergsma; "allow nagios (and everybody else) to get into swift stuff to check usage for check_disk" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2303 [15:29:34] New patchset: Mark Bergsma; "adding http monitoring to swift proxy servers" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2304 [15:29:52] New patchset: Mark Bergsma; "more db upgrades" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2305 [15:30:14] New patchset: Mark Bergsma; "wrong array" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2306 [15:30:34] New patchset: Mark Bergsma; "replaced placeholder ganglia-logtailer module with real one" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2307 [15:30:53] New patchset: Mark Bergsma; "running logtailer every minute instead of every 5 for more detailed stats. fixing typo." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2308 [15:31:20] New patchset: Mark Bergsma; "fixed inconsistent naming of the metrics" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2309 [15:31:40] New patchset: Mark Bergsma; "puts are 201, let's split them out." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2310 [15:31:57] New patchset: Mark Bergsma; "adding db1017 as an eqiad db collector" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2311 [15:32:14] New patchset: Mark Bergsma; "configuration for rolling rsyncs from dataset1001 to dataset2 and v.v." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2312 [15:32:30] New patchset: Mark Bergsma; "updated list of server for nrpe to include nagios server" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2313 [15:32:48] New patchset: Mark Bergsma; "Revert "Revert "Merge remote-tracking branch 'origin/production' into test""" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2314 [15:33:05] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2298 [15:33:05] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2302 [15:33:05] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2302 [15:33:05] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2303 [15:33:06] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2304 [15:33:06] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2305 [15:33:06] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2306 [15:33:06] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2307 [15:33:07] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2308 [15:33:07] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2309 [15:33:08] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2310 [15:33:08] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2311 [15:33:09] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2312 [15:33:09] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2313 [15:33:10] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2314 [15:33:10] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2299 [15:33:11] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2300 [15:33:11] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2301 [16:08:55] o.0 [16:32:10] hi rona [16:32:14] gggrr h [16:32:16] hi roan [16:33:11] Morning [16:33:12] can you help me in helping andre getting his git / gerrit / git-review setup working? there is something odd going on [16:33:23] Sure [16:33:45] Although if he's on Windows it's possible that I can't help him with some things [16:35:35] Andre: are you around? [16:35:45] i've got windows as backup to help [16:38:23] Am now [16:38:49] Had someone on the door trying to sell something [16:39:11] maybe you can paste your error in pastebin.com [16:39:33] Ok, one moment [16:42:42] http://pastebin.com/vfA3rarS [16:43:05] roan: can you have a look? [16:43:49] Oops, sorry, something went wrong [16:44:04] Changing it... [16:44:11] Hmm, OK, so it looks like Gerrit does not recognize your SSH key [16:44:22] Did you log into the Gerrit web interface and add your public key there? [16:44:43] ( https://labsconsole.wikimedia.org/wiki/Git#Set_up_your_ssh_key_in_Gerrit ) [16:45:13] http://pastebin.com/wAExwj5Z for the corrected version [16:45:51] Yeah, you're still getting Permission denied (publickey). [16:46:05] So check that your public key is listed in the Gerrit web interface [16:49:20] I seem to be unable to log in... [16:50:23] I thought my login would be a_engels, but combined with the password I thought I had it doesn't work [16:53:52] roan: gerrit login === labs login? [16:54:01] Yes [16:55:46] andre: but your labs login works, right? [16:56:00] Yes it does [16:56:17] are you sure you cannot login to Gerrit? [16:56:22] * johnduhart needs to send out an email about phabircator [16:56:40] And my login there is a_engels, and the password I'm trying does work for a sudo on analyticfs [16:56:47] s/analyticfs/analytics [16:57:23] But I still get http://pastebin.com/wAExwj5Z [16:57:48] If you can't log into Gerrit through the web, git is definitely gonna give you errors [16:57:58] You may have different user names for shell login and wiki logs [16:58:00] *login [16:58:47] You're also not in the Analytics group in Gerrit [16:59:02] * RoanKattouw adds [16:59:18] I suppose you can see what user name I should have [16:59:41] ? [16:59:58] I tried [17:01:04] Andre Engels (Created on 12 January 2012 at 19:14) [17:01:10] So try "Andre Engels" ? [17:01:25] Yes, Andre Engels with the suspected password works [17:01:41] Excellent [17:01:49] Now follow the instructions at https://labsconsole.wikimedia.org/wiki/Git#Set_up_your_ssh_key_in_Gerrit [17:01:59] (Those instructions aren't very clear, they should be expanded [17:03:09] You need to log into Gerrit, click Settings, click SSH Public Keys, then click Add Key [17:03:23] Yes, I am there... Now getting my key somewhere [17:03:36] Then in the text box, paste the contents of id_rsa.pub (the DOT PUB file, NOT ID_RSA WITHOUT THE DOT PUB), and click Add [17:05:18] Ok, added it [17:05:39] OK [17:05:40] WOOT WOOT [17:05:42] Now git should Just Work [17:05:44] And it also seems now clear what went wrong before [17:05:50] emphasis on *should* [17:06:27] Namely - I used the PuTTY version instead of the general one [17:06:39] Oh, you used the PuTTY format for the key? [17:07:15] on git for windows, you can either use ssh offered by git or use an external ssh client like putty [17:07:36] not sure what the easiest solution is [17:08:19] Hmmm... Still not working :-( [17:09:06] paste the new error in pastebin [17:11:00] Still the same - http://pastebin.com/g7DkE08H [17:11:29] and what if you just do ssh -p29418 gerrit.wikimedia.org [17:11:31] does that work? [17:12:08] You'll need to do ssh -p 29148 -l a_engels gerrit.wikimedia.org probably [17:12:18] Permission denied (publickey). [17:12:42] Trying Roan's command [17:14:16] ssh: connect to host gerrit.wikimedia.org port 29148: Bad file number [17:14:41] At least it's another error message :-0 [17:15:52] huh [17:16:02] Hmm, try [17:16:17] ssh -p29418 a_engels@gerrit.wikimedia.org [17:17:17] Permission denied (publickey). [17:17:19] again [17:17:32] and add -vvv [17:17:47] it could be a problem with your putty key actually [17:18:47] http://pastebin.com/tJ2gN3DM [17:20:40] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 31% free memory [17:20:40] debug1: Trying private key: /.ssh/id_rsa [17:20:42] debug3: no such identity: /.ssh/id_rsa [17:20:46] It can't find your id_rsa file [17:21:52] try the following [17:21:59] go to http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html [17:22:07] downloadd puttygen [17:22:12] Hmmm.... Then I'm afraid it's on a previous computer :-( [17:22:23] then load your putty key [17:22:27] No, it just doesn't know where to look [17:22:31] ok [17:22:35] I do have puttygen already [17:23:34] Ok, loaded the key in puttygen. Now what? [17:23:41] Try the ssh command again? [17:23:51] I'm not at all sure that ssh will know to use PuTTY as an agent [17:24:01] Or, ahm, Pageant I guess [17:24:02] Start Pageant and load the key [17:24:19] That has been done already. [17:24:47] OK [17:24:54] Does the ssh command work now? [17:25:41] Unlikely, as nothing has changed afaics [17:27:15] Hmm [17:27:29] it cannot find the id_rsa file because it's a ppk file [17:27:32] Indeed, no change [17:27:34] that's putty's format [17:27:37] Oh [17:27:42] Well it also doesn't know where to look [17:27:45] It's looking in .ssh [17:27:54] that's why you need to export it to openssh [17:27:58] $HOME/.ssh specifically, and $HOME is not set so it's looking in /.ssh [17:27:59] using puttygen [17:28:00] Which is indeed not where the key is located, as far as I know. [17:28:05] Yes, you need to export it to OpenSSH format [17:28:19] Okay, trying to put it in /.ssh [17:28:38] Then figure out the full path to the key file, and pass -i C:/Foo/Bar/id_rsa [17:28:41] O wait, that does not exist in Windows... Trying C:\.ssh [17:28:45] No, don't put it in .ssh, that won't work [17:28:53] Just find the file and pass in the path with -i [17:28:56] not sure if it it means /.shh, probably it's better to put in c:\Users\ssh\ [17:29:11] or something like that [17:32:40] andre: and? [17:33:08] Seems to not work because the key is in putty style [17:33:26] Enter passphrase for key 'C:\Users\andre\My Documents\private.ppk': [17:33:27] debug1: PEM_read_PrivateKey failed [17:33:29] debug1: read PEM private key done: type [17:33:31] debug2: bad passphrase given, try again... [17:33:42] Like Diederik said, you need to export the key to OpenSSH format [17:33:48] And then point SSH to that file [17:34:20] PROBLEM Free ram is now: CRITICAL on bots-sql3 bots-sql3 output: Critical: 5% free memory [17:34:41] Had thought PuttyGen would be able to do so for me, but apparently it does not [17:34:50] It can do that [17:34:57] But not if you don't explicitly tell it to [17:35:26] I clicked 'Save private key', and it only has .pkk (or all files) as an option [17:35:37] Ah, there [17:35:38] export [17:36:04] I have a choice between OpenSSH and ssh.com [17:36:22] OpenSSH [17:38:26] Enter passphrase for key 'C:\Users\andre\My Documents\id_rsa': [17:38:28] debug1: read PEM private key done: type RSA [17:38:30] debug1: Authentications that can continue: publickey [17:38:32] debug2: we did not send a packet, disable method [17:38:33] debug1: No more authentication methods to try. [17:38:35] Permission denied (publickey). [17:41:24] Andre_Engels: So when you put your public key into Gerrit, which file did you copypaste in? [17:42:12] The previous was public.key, this time it was id_rsa.pub [17:42:27] You put in both? [17:42:37] How did you have an id_rsa.pub before you had an id_rsa? [17:42:39] * RoanKattouw scratches head [17:43:09] maybe just start with a clean pub / private pair? [17:43:19] Hmmm... Might be from another attempt to create a pair. [17:43:49] Right [17:44:07] Grab the id_rsa.pub that you got when you exported the key just now (the one that goes with the id_rsa that you're using) and put that in Gerrit [17:45:03] It's the one that already was there [17:45:45] OK, then put in the other one too [17:45:50] Gerrit allows you to add multiple keys [17:46:04] That's the one I put in half an hour or so ago... [17:46:54] busy here... [17:47:23] O wait, I see another one here... Will be adding it too, just to try... [17:48:10] damn, i missed ryan again, was only here for ~10 mins? [17:48:16] Nope, that one is considered invalid [17:48:33] Is it in a .pub file? [17:49:56] * RoanKattouw grabs dinner [17:50:08] One of them is... But I have no idea whether it was created that way or I chose myself [17:54:12] okay,try to connect to gerrit.wikimedia.org using putty [17:54:40] putty will read your ppk and you have now both the public key in openssh and in putty's format on gerrti [17:54:51] so you should be able to connect using putty [17:55:22] if that does not work then you need to create a new pair because things are being mixed up somewhere, also remove all the public keys in gerrit [17:55:30] but first, let's try connecting using putty [17:57:07] Seems to fail... It tries to login, I then get about 10 lines of text but not enough time to start to read it. [17:57:59] when you start putty, at the bottom (Close window on exit) select 'Never' [17:58:58] actually, it sounds like it is working [17:59:13] ERROR: Interactive shell forbidden. [17:59:30] This is a restricted shell. If you need to use a full login, contact the system administrator at root@localhost [17:59:59] Or more fully: [18:00:07] login as: a_engels [18:00:09] Authenticating with public key "rsa-key-20110116" from agent [18:00:10] Linux formey 2.6.32-24-server #43-Ubuntu SMP Thu Sep 16 16:05:42 UTC 2010 x86_64 GNU/Linux [18:00:12] Ubuntu 10.04.3 LTS [18:00:14] formey is a Wikimedia Wikimedia gerrit (git) server (gerrit). [18:00:15] formey is a Wikimedia Wikimedia gerrit (git) server (gerrit::jetty). [18:00:18] formey is a Wikimedia public SVN server (svn::server). [18:00:19] 49 packages can be updated. [18:00:21] 37 updates are security updates. [18:00:23] The last Puppet run was at Mon Feb 6 17:34:31 UTC 2012 (24 minutes ago). [18:00:24] Last login: Mon Feb 6 17:56:40 2012 from a83-161-138-23.adsl.xs4all.nl [18:00:26] ERROR: Interactive shell forbidden. [18:00:28] This is a restricted shell. If you need to use a full login, [18:00:29] contact the system administrator at root@localhost. [18:00:33] so it is working [18:00:45] are you connecting to port 29418? [18:01:21] No. Will try that [18:02:44] Hi Andre Engels, you have successfully connected over SSH. [18:02:46] Unfortunately, interactive shells are disabled. [18:02:52] So that looks good, I'd say [18:03:30] okay, ssh access is now working [18:03:37] let's go back to git / gerrit [18:03:53] You mean the http login? [18:03:54] just try git-review again [18:04:25] Ok [18:06:02] Still the same... [18:06:06] Could not connect to gerrit. [18:06:07] Enter your gerrit username: a_engels [18:06:09] Trying again with ssh://a_engels@gerrit.wikimedia.org:29418/analytics/reportcard [18:06:11] .git [18:06:12] [18:06:14] We don't know where your gerrit is. Please manually create [18:06:15] a remote named gerrit and try again. [18:06:17] Traceback (most recent call last): [18:06:18] File "C:\Python27\Scripts\git-review.py", line 668, in [18:06:20] main() [18:06:22] File "C:\Python27\Scripts\git-review.py", line 618, in main [18:06:24] config['hostname'], config['port'], config['project']) [18:06:26] File "C:\Python27\Scripts\git-review.py", line 338, in check_remote [18:06:28] add_remote(hostname, port, project) [18:06:29] File "C:\Python27\Scripts\git-review.py", line 195, in add_remote [18:06:31] raise Exception("Could not connect to gerrit at %s" % remote_url) [18:06:32] Exception: Could not connect to gerrit at ssh://a_engels@gerrit.wikimedia.org:29 [18:06:34] 418/analytics/reportcard.git [18:33:08] petan: petan|wk: and... we have a spammer http://commons.wikimedia.beta.wmflabs.org/wiki/What%27s_the_overall_Proper_w... [18:33:50] johnduhart: ^^ [20:05:00] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 18% free memory [20:58:30] RECOVERY dpkg-check is now: OK on driver-dev3 driver-dev3 output: All packages OK [20:59:30] RECOVERY Free ram is now: OK on bots-2 bots-2 output: OK: 20% free memory [21:00:40] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 29% free memory [21:08:10] PROBLEM Free ram is now: WARNING on bots-2 bots-2 output: Warning: 19% free memory [21:20:30] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 19% free memory [21:32:20] PROBLEM dpkg-check is now: CRITICAL on driver-dev3 driver-dev3 output: DPKG CRITICAL dpkg reports broken packages [22:15:26] hey dee. [22:15:40] can you quickly test my test-commit? [22:22:47] sure