[03:19:06] 03/26/2012 - 03:19:06 - Creating a home directory for whym at /export/home/hadoop/whym [03:20:06] 03/26/2012 - 03:20:06 - Updating keys for whym [06:21:07] andrewbogott: thanks I'm ok for now [08:27:27] I'm having issues connecting through bastion to my new instance… the server is rejecting me for public key but it seems to have been passed through correctly (I see it with ssh-add-l )… anyone know what that could be a sign of? Something I missed when setting up the instance maybe?.... [08:56:40] Did it finish installing? [08:56:43] (Check the console log) [08:56:46] Sometimes they break [08:58:26] Damianz: It seems to have. Ends after all of the scripts with a login prompt [08:58:41] Perhaps just delete it and retry? see if it was just something being silly? [08:59:12] Possibly, if ssh-add -l shows the right key (as added under labsconsole) after the first puppet run it should let you in. [09:00:09] yeah [09:00:12] * Jamesofur tries again [12:57:23] mutante: hey, got a moment? [12:58:11] Need the shovel to get him out the carpark? [12:59:18] I don't understand, Damianz [12:59:44] We keep the bodies in the carpark out the way, they fill up the racks otherwise ;) [13:00:05] sumanah: whats up [13:00:27] * sumanah will ignore inexplicable things Damianz says [13:00:33] :P [13:01:05] You should be more random, it's fun :D [13:01:44] I'm random enough when I'm offduty. [13:02:45] http://www.kaasmuseum.nl/welcome.asp?lang=0 < mmmm cheese [13:12:04] * Damianz likes how you just delegated that reponsibility to someone else :P [13:14:02] hi Daniel_WMDE! [13:14:18] * Damianz gives sumanah a cookie then goes back to work [13:14:29] thank you Damianz :-) [13:24:47] mutante: around? [13:25:52] ^demon: hi [13:26:29] did someone receive my email related to community operation, because I received some spamblock warning [13:26:36] I got it [13:26:40] ah, ok [13:26:44] petan: you can always check the web archives. [13:26:48] true [13:26:50] I forgot [13:27:21] hopefully it would resolve the current issue we have on bots project :) [13:27:28] btw mutante we really need to fix nagios :D [13:27:46] Damianz: you know python? [13:28:00] I do, python is one of my fav languages. [13:28:05] I need to create a line to make thread sleep for 1 second [13:28:13] I guess sleep 1 :) [13:28:26] it would be cool to implement it to echo bot [13:28:37] so that it doesn't die when nagios post 1000 warnings at same time [13:28:40] time.sleep(1)? [13:28:44] yes [13:28:45] ok [13:29:15] now I need to find it in puppet [13:30:44] petan|wk: see #huggle [13:33:22] sure [13:41:03] !ping [13:41:03] pong [13:44:46] <^demon> petan|wk: Pong [13:45:17] ^demon: I emailed wikitech-owner regarding some email [13:45:26] <^demon> I'm looking for him now :) [13:45:28] I don't really know if that email is broken or it's just antispam [13:45:51] maybe it's sent to everyone who post a message to wikitech, I don't know [13:46:02] I sent 2 messages there today I received it after [13:46:57] <^demon> I can't find any such user. [13:48:46] <^demon> petan|wk: Maybe Abigor beat me to it. [13:54:00] hm... [13:54:11] ok [13:56:17] petan|wk: sry, I#m busy, but about the "sle and Nagios checks.. see http://nagios.manubulon.com/traduction/docs14en/checkscheduling.html "Service Interleaving" .."service_interleave_factor" "inter_check_delay_method".. i think thats the direction at least [13:56:36] arr, and my connection is so bad, random lettes are missing there [13:56:50] not really, there is nothing on nagios side what needs to be done [13:56:51] i meant to say" about the sleep 1 and Nagios checks" [13:57:06] it's all in puppet what needs to be done [13:57:13] and I have no access over that [13:57:30] nagios works fine, but I need puppet to send snmp pings [13:57:32] to nagios [13:57:40] when puppet run [13:57:58] thats a different story.. you said something about the bot dying when all checks turn CRIT at the same time [13:58:24] that I can solve if I find where is source code [13:58:42] maybe i didnt get the whole story,, it appeared like you want it to stop checking everything at once.. nevermind then [13:59:26] no, that's ok [13:59:33] there are two problems: bot and puppet [13:59:37] I need you only for puppet :) [13:59:46] bot is something else [14:00:11] speaking about bot... it's gone [14:00:28] which isn't necessarily bad give to amount of alerts we have right now [14:02:07] chrismcmahon: hi [14:02:25] I received the email from Mark today, he said that you want to set up svn up on beta? [14:03:28] petan|wk: I don't think so. what do you mean by "SVN up on beta"? [14:03:54] git > svn [14:04:20] Chris is interested in using the beta cluster for automated testing of [14:04:22] svn HEAD. Could you help him get this set up? [14:04:24] petan|wk: what I would like to see on beta is a predictable deployment from the current branch under development to beta (via Jenkins if possible) [14:04:51] ah [14:05:12] petan| hashar and I discussed this at one point, but it didn't make sense to do it until the git migration was over and stable [14:05:15] I don't know who is jenkins person... maybe ^demon knows? [14:05:40] right [14:05:42] petan|wk: hashar is the Jenkins person, and he knows about the idea [14:05:53] I don't know if migration is fully done [14:06:14] we really need to update het system now [14:06:27] so that we can keep the branches up [14:06:37] last checkout was done before migration [14:06:47] petan|wk: in the meantime, I just returned from a conference about browser test automation, and I have some things I want to try. this is a long-term project, and I don't mind if it moves slowly [14:06:59] ok [14:07:23] petan| it is more important that it be done well than that it be done quickly :) [14:07:30] :) [14:07:35] Hack, done is better than perfect! [14:07:53] depends [14:07:59] :P [14:08:17] Damianz: Most of the time I agree with that. [14:09:17] hi :) [14:09:57] petan|wk: do you know if we have Swift on beta labs yet? The last I heard that was coming, but not yet done. [14:09:58] petan|wk: I am indeed the Jenkins dude :-) [14:10:23] ok, chrismcmahon I don't think so [14:10:41] there is some in puppet I guess [14:10:51] no idea if it work [14:10:54] :o [14:11:04] I never tried it [14:11:39] petan|wk: there is a project for Timed Media Handler that we would like to test on beta/commons, and it is my belief that we should have Swift there for a valid test of TMH [14:12:06] petan|wk: I think maplebed had that as a TODO [14:12:21] chrismcmahon: I personally wouldn't block TMH testing on Swift [14:12:39] yes I think tmh is working there [14:12:44] It should be tested with Swift, but there's probably enough to be tested that doesn't have anything to do with Swift [14:12:52] there is instance configured by someone, who work on that [14:13:01] it's possible swift is on that [14:13:01] * Damianz checks hashar's status and declares his code coverage isn't enough [14:13:18] Damianz: which coverage ? :-D [14:13:29] Javascript :D [14:13:33] Need moar ajax [14:13:42] yeah [14:13:51] I have been so busy with 1.19 / git [14:14:16] I will try to finish up my Jenkins jobs rewrite this week [14:25:12] what if I want my project accessible from external network? [14:25:57] liangent: You'd have to request a public IP. We don't have many of those so you may or may not get it [14:26:40] only http access is ok. can it be done with host-header? [14:26:59] liangent: yes, Ryan is going to set up a proxy for it [14:27:07] but it may take a while for it to happen [14:27:23] @search tunnel [14:27:24] Results (found 1): putty, [14:27:39] @search bastion [14:27:39] Results (found 4): sudo, bastion, ryanland, socks-proxy, [14:27:45] !socks-proxy [14:27:45] ssh @bastion.wmflabs.org -D ; # [14:27:53] liangent: this may help you too [14:28:43] petan|wk: so the visitor must have a labs account first? [14:28:54] depends [14:29:12] Yes, they need SSH access to labs in order to use the tunnel [14:29:20] if you want to use socks proxy, then it could be problem without account [14:29:35] however Ryan wanted to have a shared public ip [14:29:40] which would be behind nat [14:29:50] so that projects which require like 1 port only could use it [14:29:57] yeah [14:50:38] http://commons.wikimedia.beta.wmflabs.org/wiki/Main_Page is taking a very long time to return for me, on the order of minutes. known issue? [14:56:57] it loaded to me 5 sec [14:57:04] no ide [14:57:05] ide [14:57:08] idea * [15:14:37] hashar: what's ur email? [15:14:42] I want to put u in cc [15:14:55] amusso@wikimedia.org for anything related to projects [15:15:12] hashar@free.fr is my spam mail for notifications, probably not the one you want :-] [15:19:42] petan, we were being crawled by google a few days ago [15:20:05] the symptoms were very much like that, "site being slow" [15:20:17] apache reached maxchildren [15:20:44] is there any point in having labs wiki crawled ? [15:21:00] if not, you might fix that using a robots.txt [15:21:14] well, robots were addiionally producing a 500 error [15:21:20] I managed to fix that, though [15:22:14] I didn't search for the location of the master robots.txt file [15:33:02] Platonides: I think we could just make it nobots [15:33:20] no reason to index [15:53:45] +1 for no robots, no need to slowdown the test instance serving googlebot [16:06:50] Should I expect a long wait for a git repository? [16:07:47] marktraceur: What do you mean? Did you request a new repository? [16:08:03] Yeah, but it looks like the backlog is pretty huge [16:10:03] Link? [16:10:47] RoanKattouw, http://www.mediawiki.org/wiki/Git/New_repositories/Requests [16:11:24] Right [16:11:29] Do you have any existing code to import? [16:11:36] If so, was that code managed in git or SVN? [16:11:43] Yeah, but I can do it myself through git [16:11:49] The repository is on my own machine [16:11:52] Oh, OK [16:11:56] Then it should be easy [16:12:00] I just need somewhere to push [16:12:04] A bunch of those have to be imported from SVN [16:12:13] Yuck! [16:12:45] Is it a MediaWiki extension? [16:12:57] No, it's a separate software [16:13:23] How is it related to Wikimedia? [16:13:45] Well, it's Wikimedia's software, I'm making it for Erik Moeller :) [16:13:57] OK [16:14:13] I wasn't aware of that :) [16:14:17] Indeed [16:14:29] orgcharts, by the way, is the name [16:14:53] * RoanKattouw would create a repo for it but is unsure where to put it [16:14:59] What kind of software is it? What does it do? [16:15:20] It's a web application, it shows employee information [16:15:37] Opps [16:15:42] Wrong key [16:15:44] http://orgcharts.wmflabs.org:8888/ [16:15:47] Hmm, I'm gonna defer to ^demon|away as to where to put it [16:15:56] All righty then [16:16:40] (BTW, some of the data in that example chart is inaccurate) [16:17:01] Erik has been playing with it, it's not meant to be perfect [16:17:12] OK [16:17:12] He's trying to break it [16:17:19] Been pretty successful so far :) [16:17:22] Well it's not your fault that Erik doesn't know where certain people live :) [16:17:31] I can fix it if you'd like [16:18:15] heh, no worries, I'm just a very pedantic person [16:18:41] Fair enough [21:23:06] Hey alejrb :) [21:23:08] heya [21:23:17] Ok, lets see what we can do for you [21:23:25] awesome [21:24:47] alejrb: Ok, first, pop over to http://www.mediawiki.org/wiki/Project:Labsconsole_accounts and Request an account [21:24:56] OK, will do [21:25:48] As soon as you get an account, I can make you a server instance of your own and then we will be good to go :) [21:27:08] cool [21:27:14] how long do requests usually take to process? [21:28:34] Not sure, I'm wondering where Ryan is... [21:28:49] mmk [21:29:09] But the last was done 14:45, 26 March 2012 (UTC) [21:29:14] So it might be tomorrow [21:29:44] I'll get the server made now, and add you to it as soon as you get the account [21:29:54] ok, great [21:29:55] thansk [21:31:18] bots-gloo i-000001d1 pending m1.small [21:33:09] !log bots Created bots-gloo for alejrb for IGLOO [21:33:21] Damn it morebots [21:36:43] methecooldude: what's your svn id? [21:36:51] RichS ? [21:36:53] I think [21:37:08] ah okay, I thought I'd not seen "methecooldude" before :) [21:37:14] Thehelpfulone: ^_^ [21:37:34] it's "Rich Smith" [21:37:40] for labs I meant* [21:37:47] Ah, ok [21:54:55] alejrb and/or methecooldude: Ryan is out for the week... I should be able to do at least a small subset of what he does here. [21:55:05] But, as for account creation... lemme see if Sumanah is about. [21:55:21] andrewbogott: Thanking you [22:04:48] alejrb: I believe that I just created an account for you. Can you check your email and see if things are working? [22:04:57] * andrewbogott is still learning how to do this. [22:04:58] sure, hang on [22:05:34] yeah, I got the email confirmation [22:06:18] thanks [22:13:21] alejrb: Does it actually grant you access and such? [22:13:53] oh, yeah, I logged on [22:14:47] So I believe so [22:16:35] alejrb: you need to add your public SSH key in [22:16:53] see https://labsconsole.wikimedia.org/wiki/Help:Access#Initial_log_in [22:17:07] but go straight to step 4 as your password was emailed to you alejrb [22:17:17] it needs to be in an open SSH format [22:18:34] then anyone can add you to bastion [22:18:56] and if you want bots access, I can give you that as can methecooldude [22:19:18] Thehelpfulone: It instance is done, was just waiting for the user [22:19:28] OK, I added my key [22:20:13] 03/26/2012 - 22:20:12 - Creating a home directory for alejrb at /export/home/bastion/alejrb [22:20:41] ok so I added you to bastion, methecooldude can do bots - although I thought your key adding would show up here too.. [22:21:09] 03/26/2012 - 22:21:09 - Creating a home directory for alejrb at /export/home/bots/alejrb [22:21:11] 03/26/2012 - 22:21:11 - Updating keys for alejrb [22:21:37] Thehelpfulone: There is it ^ [22:21:49] yep [22:22:09] 03/26/2012 - 22:22:08 - Updating keys for alejrb [22:22:22] great :) [22:29:40] alejrb: Ok, have you read how to log in? [22:30:46] still working on it [22:30:53] putty is owning me [22:32:13] alejrb: Why, what's it doing? [22:33:18] rejecting the key file [22:33:24] Ooer [22:34:05] so I'm trying to connect to alejrb@bastion.wmflabs.org ? [22:34:23] is that right? [22:34:35] alejrb: Yea [22:37:27] it's saying to supported authentication methods available. [22:37:33] no* [22:37:35] thoughts? [22:37:44] the key is OpenSSH SSH-2 private key [22:37:51] Sounds like PuTTy isn't talking to Pagent [22:37:56] hmm [22:38:47] Let me just re-install PuTTY and I'll check stuff [22:40:40] alejrb: Check SSH > Auth and make sure "Attempt authentication using Pageant" [22:41:11] is on [22:41:14] oh, got on [22:41:30] I was trying to use the non-putty version of the key -.- [22:41:45] Ahh [22:42:53] OK, so I have the bastion prompt. Do I need to do something else? [22:43:17] alejrb: ssh bots-gloo [22:43:24] WAit, ignore me [22:43:35] I'll make you a PuTTY config, hold on [22:43:40] oh, awesome [22:46:14] alejrb: Give me a few moments [22:46:22] no problem [22:47:35] you need to ssh -A bots-gloo [22:47:54] you have to get into bastion first, then type that when you're in bastion [22:48:53] Thehelpfulone: Does -A sent the key along? [22:48:56] send* [22:48:59] yes [22:49:17] alejrb: There we go then, see Thehelpfulone :) [22:50:28] hmm, I get Permission denied (publickey) [22:50:55] is that my config, or did I miss something on the server? [22:51:13] alejrb: Did you add your PuTTY key on labsconsole? [22:51:31] alejrb: Are you a member of the project that contains the host bots-gloo? [22:51:37] andrewbogott: Yes, he is [22:51:39] I added him [22:51:42] 'k [22:51:57] lemme check [22:52:58] your putty key needs to be in open SSH alejrb [22:53:16] alejrb: Did you create it using PuTTYGen? [22:54:00] so starting like ssh-rsa as opposed to ---- BEGIN SSH2 PUBLIC KEY ---- [22:54:01] No, I had an open SSH key, then I used puttygen to convert it so that it works with pagaent [22:54:20] so the public key in the labs console, which is letting me onto bastion, is in that form [22:54:26] alejrb: Ah, ok [22:56:04] alejrb: I would use PuTTYGen to create a now SSH-2 RSA key, then import that to labsconsole, add it to Pageant (and remove the old one) and try that [22:56:50] kk [22:57:06] I'm looking at https://labsconsole.wikimedia.org/wiki/Git#Checking_out_the_repositories [22:57:17] it lists which branch is used in production and which in the 'testlabs' project. [22:57:22] what's used in all the rest of the projects? [22:58:26] i.e. I want to do something in my new project. what should I check out to start mucking with stuff that will affect my project? [22:58:46] maplebed: You're talking about puppet stuff? [22:58:53] yes. [22:59:05] Tragically, there is only the test branch to work with at the moment. [22:59:09] 03/26/2012 - 22:59:09 - Updating keys for alejrb [22:59:12] 03/26/2012 - 22:59:12 - Updating keys for alejrb [22:59:19] so at that link, the test branch says "used in the testlabs prject". [22:59:22] does that lie? [22:59:27] is it actually used in all projects? [22:59:49] (hmm. I'm making an inference here. "used in the testlabs project "... and not in any other project.) [23:00:10] 03/26/2012 - 23:00:09 - Updating keys for alejrb [23:00:12] would it be better to say "used to configure labs instances"? [23:00:13] 03/26/2012 - 23:00:13 - Updating keys for alejrb [23:00:19] maplebed: As far as I know, it lies. I have definitely made changes in the 'test' branch and seen them take effect in the openstack project, for example. [23:00:35] ok. I'm going to make that change. someone can revert me if it's wrong. [23:00:48] I think that the status quo is known to be so broken that no one wants to spend even a second documenting it. [23:01:02] yeah, well, I'm trying to start up a new project [23:01:06] and stabbing in the dark. [23:01:15] there will be more like me before it is "fixed", so... [23:01:16] :P [23:01:21] Someday soon each project will have its own branch. [23:01:30] RSN(tm). [23:02:05] and we'll be able to do puppet runs on a local repo so you can test before you push! [23:02:09] 03/26/2012 - 23:02:09 - Updating keys for alejrb [23:02:09] It's a beautiful dream. [23:02:12] 03/26/2012 - 23:02:12 - Updating keys for alejrb [23:02:43] Really, I think ryan is going to create per-project branches, like, next week or something. [23:03:40] maplebed: If you want to create new classes and variables that are specific to your project, there's actually a way to do that now, sort of. [23:03:50] I think I found that. [23:04:03] https://labsconsole.wikimedia.org/wiki/Special:NovaPuppetGroup ? [23:04:18] * andrewbogott nods [23:04:38] That changes what options show up in the gui. I suspect it does not change what actually acts on the hosts though. [23:05:51] I think I need to go through it though, [23:05:56] because none of the swift stuff exists in the gui. [23:06:09] sure. [23:08:59] methecooldude: So I tried that, and it will still let me into bastion, but not into bots-gloo [23:09:46] Are you using ssh -A bots-gloo ? [23:10:16] yes [23:10:22] Urm, ok [23:10:27] Goive me a moment [23:10:34] Giuve* [23:10:36] .. [23:10:38] Give* [23:10:40] haha [23:12:33] Heh... PuTTY configs can't be sorted in a file... [23:13:56] alejrb: Anyway, start up a new PuTTY and go to SSH > Tunnels, in source port put a random port of your choice, and in Destination put bots-gloo:22 [23:14:05] Then make a bastion connection with those settings [23:14:26] Then open a new PuTTY but connecting to localhost:port you choise [23:21:17] methecooldude: It says no supported authentication methods available [23:21:41] alejrb: Urm,. that'd odd casue that's the way I connect to bots-cb [23:21:55] I have the bastion connection forwarding 8085 to bots-gloo:22 [23:22:07] and the putty connection to localhost:8085 fails [23:22:31] andrewbogott: I don't know if you have root on bastion but it's saying " There are 2 zombie processes." [23:23:19] alejrb: Ok, lets see what is going on here [23:23:36] yep :) [23:36:09] 03/26/2012 - 23:36:08 - Updating keys for alejrb [23:36:12] 03/26/2012 - 23:36:11 - Updating keys for alejrb [23:40:30] What the failboat... [23:40:36] ? [23:40:51] alejrb: bots-cb works fine... bots-gloo doesn't [23:41:06] well that would explain why it's not letting me on lol [23:41:29] alejrb: Ok, we will do this the other way... do you want to run it on bots-cb, the only other people who use it are me and Damianz for ClueBot [23:41:50] I don't really mind where it goes [23:41:53] but an important point [23:42:04] is that users have to be able to connect to it as a webserver [23:42:07] is that how this works? [23:43:03] alejrb: You put the webserver stuff in /mnt/public_html/alejrb and bots-apache1 handles it with URL of http://bots.wmflabs.org/~alejrb [23:43:29] I don't mind how it works, as long as there's a URL I can put in the igloo javascript :) [23:43:55] alejrb: Yep, and it also means you can remove the non-Wikimedia warning :P [23:44:04] Becasue, it will be at Wikimedia :P [23:44:11] lol [23:44:26] In which case I don't mind what instance it runs on, it should make no difference [23:44:36] it also needs a database [23:44:37] Anyway, change your PuTTY tunnel to bots-cb:22 [23:44:40] kk [23:45:32] It should let you in on your public key [23:45:55] yep, let me in right away [23:46:15] phew, I thought I was making a stupid mistake :D [23:46:19] Cool :) [23:46:26] No, it's the instance being stilly [23:46:28] silly* [23:46:32] lol [23:48:55] Anyway, while your bastion session is up, you can use WinSCP (or related program) to transfer files, just connect to localhost:port [23:49:12] ok, great [23:49:32] Hope to see Igloo back up soon :D [23:49:40] I can manage SQL through that also? [23:50:15] alejrb: I'm not sure about SQL... I'll ask Ryan about that when I see him next [23:50:31] I can change it to use flat files... :P [23:50:35] Unless andrewbogott is still around and knows what's going on with MySQL [23:51:00] alejrb: There is a SQL instance for bots... but I'm not sure if t's working yet [23:51:17] hm, ok [23:51:46] alejrb: I might be able to poke a MySQL onto bots-apache1 if you need it [23:52:17] well atm it uses a database to store everything it uses, so it needs one to run [23:52:32] Ok, let me see if I can get one going [23:52:46] there's no especial hurry though if there's a reason it's switched off [23:56:37] alejrb: The joys of having sudo :D [23:56:43] lol [23:57:18] methecooldude: I'm still here, but I don't know anything. [23:57:23]