[00:02:27] The instance creation blank page is fixed. [00:02:40] ah. cool [00:02:44] what was the issue? [00:02:53] also, are arecords being added? [00:03:22] yep. arecords are being applied [00:04:44] hosts for new instances were being created as public rather than private hosts. [00:04:50] oh. weird [00:04:58] I don't understand why that wasn't glaringly obvious on nova-precise2 though [00:05:03] heh [00:05:13] Well, I guess, actually -- the ldap records were getting created as private. [00:05:21] It was just the ephemeral host object that was wrong. [00:05:33] So it only mattered for the split second after creation and before the page finished. [00:05:55] oh. ok [00:06:09] how was the ephemeral being created incorrectly? [00:06:27] https://gerrit.wikimedia.org/r/#/c/86208/ [00:06:42] :D [00:06:43] I see [00:06:45] yep! [00:08:11] ok. I sent an update to the list [00:11:12] looks good -- are you still fixing certs or is that done already? [00:14:42] still fixing them [00:21:18] Ryan_Lane, need to go feed fish and self, but I'll keep laptop in earshot -- page me if you find other broken things. [00:21:25] * Ryan_Lane nods [00:21:40] we're going to head off for food too. I think everything except puppet is good now [00:22:01] and that should be fixing itself [01:57:01] [bz] (8RESOLVED - created by: 2Yuvi Panda, priority: 4Unprioritized - 6normal) [Bug 54636] wikitech.wikimedia.org needs to have VE enabled - https://bugzilla.wikimedia.org/show_bug.cgi?id=54636 [01:57:13] Ryan_Lane: ^ the pref seems to be set to 'off' [01:59:52] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Migration of Toolserver tools was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=790949 edit summary: /* How will Toolserver tools work on Wikimedia Labs? */ help link [02:20:47] hmm [02:20:53] I get permission denied for my public key [02:20:57] when logging in via bastion?!? [02:34:28] Ryan_Lane: Coren getting 'Connection closed by UNKNOWN' when trying to ssh to a new instance [02:35:00] Are you a member of the project this instance is in? [02:35:30] Coren: indeed [02:35:36] What instance? [02:35:39] Coren: bd808 just created the instance, and is getting the same error [02:35:41] Coren: bd808-test [02:35:45] in project multimedia [02:36:22] ... you do realize it takes quite a while before puppet runs and completes when first creating an instance, right? [02:36:37] Coren: usually it says 'RUNNING' or similar in the interface when that happens [02:36:47] before now I've always been able to log in when it says 'ACTIVE' [02:37:23] That just means the instance has been booted; it may take >15 min before puppet runs enough time for the box to be valid; sometimes more when the apt update takes a bit. [02:37:38] ah. reasonable [02:38:07] And that may fail completely if you configured any puppet class when creating it (hence the warning on that page) [02:38:14] heh [02:38:27] we should hide that with CSS or something [02:38:37] 'hey! look at this! don't do anything here, okay? move along...' [02:38:42] Coren: also, have you seen https://wikitech.wikimedia.org/wiki/Labsvagrant? [02:41:12] Coren: I'm not able to log in to proxy-project-proxy either [02:41:15] Coren: says 'public key denied' [02:41:25] Coren: which... is weird? since I was logged in there just a bit before [02:41:30] and I'm part of the project [02:41:31] admin even [02:41:37] That /is/ wierf [02:41:41] wierd* [02:42:06] this is instance proxy-project-proxy on project project-proxy [02:42:11] Hmmm. [02:42:14] let met ry another [02:42:17] *let me try another [02:42:24] oh hmm [02:42:28] Coren: i can login to other instances [02:42:30] on that project [02:42:39] I wonder if it is labsvagrant messing things up [02:43:06] labsvagrant doesn't really have anything for ssh [02:43:09] Coren: can you login with root key? [02:43:20] I suppose it's possible; but I can't log in to those instances with the root key either so it's nasty. [02:43:31] so that instance is pretty much, uh, lost? [02:43:43] * Coren idly wonders if Ryan_Lane's DNS changes might have something to do with that. [02:43:48] Ryan_Lane: ^^ [02:44:14] it is just *one* instance tho [02:44:28] Well, two. [02:44:40] they have *different* errors [02:44:47] one says 'Closed by UNKNOWN' [02:44:48] But everywhere else I semi randomly test works. [02:44:51] other says something else [02:44:52] yeah [02:44:52] hmph. [02:45:07] Coren: why can't you login with root key? [02:45:21] bd808-test just gives me permission denied [02:45:28] you aren't ont he project [02:45:30] wait [02:45:36] permission denied means that sshd is u [02:45:37] p [02:45:41] YuviPanda: I can't very well go and check the logs if I can't log in to that box. [02:45:48] I know ssh is up. [02:45:49] indeed [02:46:13] * Coren ponders. [02:46:14] Coren: added you to multimedia [02:46:29] and admin [02:46:29] That won't change root key access. :-) [02:46:32] right [02:46:43] oh wait [02:46:50] you said you can't login to *those* two instances with root key [02:46:52] okay [02:46:54] i misunderstood [02:46:55] nevermind [02:46:57] maeks sense now [02:48:28] bd808: of course, there is the usual step 3.1 of 'wait, this part of labs seems to be slightly broken!' [02:48:47] http://dpaste.de/19AJp/ [02:49:07] pretty much the same here [02:49:26] Coren: I created a new instance called silver-hammer, same output as bd808-test [02:49:37] How long ago? [02:49:44] bd808: clearly sshd is up, but somehow connection is closed. Is that raelly a puppet issue? [02:49:53] Ryan_Lane: andrewbogott_afk: You have teh broken something. [02:50:19] YuviPanda: ssh config comes from puppet, including things like authorized groups etc. [02:50:24] YuviPanda: I.e.: yes. [02:50:30] sure, but wouldn't that be a publickey denied? [02:50:36] rather than a UNKNOWN [02:52:01] Coren: can you say /what/ is broken? :D [02:52:32] let's blame glusterfs [02:52:34] :P [02:52:37] new instance creation, for puppet runs? [02:52:43] YuviPanda: we made a giant dns change today [02:52:58] yeah, i read the backscroll [02:53:09] new instance creation is probably broken, yeah [02:53:14] s/probably/potentially/ [02:53:24] what happens? [02:53:34] Ryan_Lane: http://dpaste.de/19AJp/ [02:53:49] Connection closed by UNKNOWN [02:55:01] is this a lucid or precise image? [02:55:10] precise [02:55:12] and was it just created? [02:55:18] yeah [02:55:22] ok [02:55:24] 12.04, just created about 10 minutes ago [02:55:30] I just created an instance, to double check. [02:55:31] oohhhh [02:55:32] crap [02:55:33] sorry [02:55:35] Too early to judge though [02:55:47] All defaults on the creation form [02:55:48] created 2013-09-27T02:47:32Z [02:55:50] totally my fault [02:56:06] I had disabled the puppet-signer cron while I was fixing the puppet issues [02:56:09] and didn't re-enable it [02:56:15] so puppet wasn't signing certs [02:56:22] which means no puppet run [02:56:25] ah. that would be a problem [02:57:04] would that also explain why proxy-project-proxy gives me a permission denied? [02:57:10] So those instances will revive in 30? [02:57:14] YuviPanda: no [02:57:22] andrewbogott: new ones? yeah [02:57:36] though it's weird that root ssh isn't working either [02:57:43] Ryan_Lane: heh, any idea why proxy-project-proxy gives me a permission denied? :D [02:57:44] that should work immediately when the instance boots [02:57:46] Ryan_Lane: the instance is still up [02:57:53] and it's like, months old [02:57:57] hm [02:57:59] and I was ssh'd in before I went to sleep [02:58:03] it's also giving me permission denied [02:58:27] YuviPanda: you broke it with dream hacking! [02:58:30] I was trying out labsvagrant stuff on it - so it *might* be labsvagrant - but I doubt it, since the vagrant puppet files don't touch sshconfig [02:58:38] and it was working fine before that [02:58:38] so [02:58:53] bd808: shushhh, no broadcasting my secret :P [02:59:13] * bd808 knows that YuviPanda knows there is no spoon [02:59:27] there's no chopstick [02:59:28] only fork [02:59:40] and spork [02:59:54] YuviPanda: is that the proxy host that actually handles the proxying? [02:59:58] Ryan_Lane: nope [03:00:01] Ryan_Lane: that's proxy-dammit [03:00:04] right [03:00:12] have you tried rebooting? [03:00:14] Ryan_Lane: this is my 'test' host [03:00:17] does the vagrant stuff modify the ssh config? [03:00:21] nope [03:00:48] it pokes MOTD [03:00:53] try rebooting ;) [03:00:54] is the closest to ssh it does anything [03:00:59] maybe you OOM'd it or something [03:01:04] does it set a robots.txt? [03:01:14] rebooting [03:01:16] does it do the other things the labs wiki role does? [03:01:19] Ryan_Lane: the *instance* is up [03:01:27] since I could hit the webserver via instanceproxy [03:01:29] we did a whole bunch of crap that legal wanted us to do [03:01:55] Ryan_Lane: define 'it'. dynamic proxy doesn't do robots.txt special handling, no [03:02:08] no, I mean the vagrant stuff [03:02:20] hm. this new instance surely isn't working... [03:02:35] that makes no sense... [03:02:46] Ryan_Lane: vagrant stuff was just me testing at the closest available self puppet thing. has nothing to do with the proxy project [03:03:19] Ryan_Lane: ah, and a robots.txt for labsvagrant makes sense, although I wonder if we should handle that at the proxy level itself? [03:03:36] it's working in puppet [03:03:56] what is 'it'? [03:03:58] robots.txt? [03:04:02] or new instance? [03:04:04] YuviPanda: well, the labs role does all kinds of stuff [03:04:14] I meant this instance I'm testing, sorry ;) [03:04:14] *which* labs role? [03:04:16] Ryan_Lane, my 10-minute-old instance is working fine, personal and root key both. [03:04:18] ah, right :) [03:04:22] Ryan_Lane: rebooting bd808-test fixed ti [03:04:33] andrewbogott: hm. I guess maybe the image doesn't work like it should any more :( [03:04:36] s/ti/it/ [03:04:42] Ryan_Lane: proxy-project-proxy rebooted, no luck yet. [03:04:43] that sucks. [03:04:47] how so? [03:04:51] YuviPanda: sometimes it takes a bit [03:04:56] * YuviPanda reboots silver hammer [03:05:10] andrewbogott: the image itself is configured to allow immediate login [03:05:35] oooohhhhhhh [03:05:40] project=`ldapsearch -x -D ${binddn} -w ${bindpw} -b ${hostsou} "dc=${id}" puppetvar | grep 'instanceproject' | sed 's/.*=//'` [03:06:05] Ryan_Lane: okay, rebootign fixed silver-hammer too [03:06:44] what, doesn't let me sudo!? [03:08:52] Ryan_Lane: hmm, I can sudo fine on multimedia-dragons, but not on silver-hammer. same project, silver-hammer is new [03:09:06] YuviPanda: yeah, something was broken [03:09:07] how new? [03:09:13] do a puppet run and try again [03:09:19] Ryan_Lane: can't do puppet run, can't be root [03:09:23] :D [03:09:27] Ryan_Lane: about 10 mins :P [03:09:31] 'be root so you can be root' [03:09:43] I'm doing a run [03:09:52] elitist! [03:09:58] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Role::Labsvagrant] is already defined; cannot redefine at /etc/puppet/manifests/role/labsvagrant.pp:5 on node i-000008f9.pmtpa.wmflabs [03:10:04] did you build it with that class? [03:10:40] Ryan_Lane: I just added it [03:10:41] in config [03:10:48] didn't build with it [03:10:50] remove it [03:11:01] doing [03:11:03] also weird error [03:11:16] well, maybe the first puppet run didn't go through? [03:11:19] removed [03:11:28] Should sudo prompt me for a password on a labs instance? (bd808-test) [03:11:46] bd808: no [03:11:47] one sec [03:11:59] * bd808 thought that was fishy [03:11:59] bd808: same issue I'm having, Ryan_Lane's looking into it [03:12:07] YuviPanda: try now [03:12:10] * bd808 waits patiently [03:12:21] bd808: please remove the vagrant class from the config [03:12:38] Ryan_Lane: still asks for password [03:12:55] Ryan_Lane: {{done}} [03:13:44] which project is this? [03:13:51] Ryan_Lane, are you already forcing a puppet run on silver-hammer? [03:13:56] tools? [03:13:57] If not then I can, I'm logged in as root. [03:14:03] andrewbogott: yeah [03:14:04] multimedia project; bd808-test instance [03:14:05] I did already [03:14:08] ok [03:14:38] same project as silver-hammer FWIW [03:15:31] silver-hammer is multimedia? [03:15:33] or tools? [03:15:42] multimedia [03:16:03] one sec [03:16:13] hm. weird [03:16:30] ah [03:16:47] delete these instances [03:16:49] they're broken [03:17:27] any instance built since the dns change is [03:17:36] delete bd808-test: {{done}} [03:17:36] till about 5 mins ago [03:18:06] yeah, seems fixed now [03:18:10] yep. is [03:18:11] let me delete mine [03:18:12] too [03:18:43] there's a script that runs when the instance is created and it changes come canned values in config files on the image [03:18:57] it was broken, and all of the config files got something set incorrectly [03:19:31] right [03:19:50] Ryan_Lane: thanks! new instance came right up with ssh and sudo working [03:19:54] great [03:20:03] Coren: ^^ see! :D [03:20:34] so much shit to test when a large change is done :( [03:20:38] hehe [03:20:44] now on to step 4 in my plan to learn how to use labs [03:20:47] :D [03:20:57] Ryan_Lane: any idea of why proxy-project-proxy gives me publickey denied? [03:22:01] I knew it was Ryan's fault. :-) [03:22:40] Oh, and by the way: I called it back in March: http://store.steampowered.com/livingroom/SteamMachines/ [03:23:26] Ryan_Lane: also any idea about the 'duplicate definition of role::labsvagrant' error? [03:24:56] another n00b question: bd808-test seemed to have my same homedir as multimedia-dragons, but new multimedia-bd808 does not [03:25:13] SteamOS: Linux tuned for gaming with steam on top. Q: "Can I download the OS to try it out?" [03:25:13] Which is "normal" [03:25:13] A: "You will be able to download it (including the source code, if you're into that) but not yet." [03:25:21] Gabe Gets It [03:27:25] bd808: so [03:27:32] bd808: by default the instances do not get NFS [03:27:32] pearl-hammer (new instance YuviPanda just created) also has my homedir [03:27:34] bd808: but get glusterfs [03:27:44] bd808: homedir is on either glusterfs or nfs [03:27:54] bd808: but glusterfs sucks, so all the multimedia-* stuff is on nfs [03:28:15] then step 4 is "get nfs homedir" [03:28:16] bd808: you can enable the ' role::labsnfs::client ' to enable nfs [03:28:24] bd808: yup. I don't know why it isn't default, tho [03:28:28] Ryan_Lane: Coren ^ [03:29:01] bd808: get nfs homedir is 1. enable role, 2. run puppet with 'sudo puppetd -tv', 3. restart instance [03:29:28] It's not the default because we aren't confident about the hardware issue yet; I'm currently redoing labstore4 with the older driver and (obviously) different hardware. If that plays well, we'll probably start migrating en masse. [03:32:06] YuviPanda: step 4 {{done}}, now to try your new role [03:32:16] bd808: did you check it mounted properly? [03:32:18] mount? [03:32:34] homedir is there now [03:32:49] reboot was practically instant! [03:32:59] weee! [03:34:35] Ryan_Lane: ! [03:34:36] Ryan_Lane: info: Creating a new SSL key for pearl-hammer.pmtpa.wmflabs [03:34:41] Ryan_Lane: err: Could not request certificate: getaddrinfo: Name or service not known [03:34:44] Ryan_Lane: Exiting; failed to retrieve certificate and waitforcert is disabled [03:35:02] Ryan_Lane: oh, don't mind me. apparently that happens if you forgot to become root [03:36:02] Coren: I wonder if 'err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Role::Labsvagrant] is already defined; cannot redefine at /etc/puppet/manifests/role/labsvagrant.pp:5 on node i-000008fe.pmtpa.wmflabs' is a side effect of how that role was added to the interface? [03:37:06] Well, that only allows adding the role to the node definition through the web interface. If you check it there /and/ it's included elsewhere... :-) [03:37:53] Coren: it isn't included anywhere! [03:37:56] else [03:38:08] Coren: it is defined there [03:38:22] define != include, right? [03:38:29] YuviPanda: I get the same dup def error when I enable the labsvagrant role via web interface [03:38:38] bd808: I enabled it via the web interface only [03:38:53] * Coren goes to bed. Have fun debugging. [03:38:54] :-) [03:39:11] bd808: and hence we have to wait for tomorrow, since nobody awake now has the rights to debug this issue [03:39:13] grrr [03:39:20] timezones suck [03:39:28] and being awake during the day seems to be completely unworkable for me [03:40:06] Ryan_Lane: you around? [03:40:21] I have "issues" with that too. My brain mostly is creative when I shouldn't be working anymore. [03:41:31] bd808: jet lag has me waking up at 5am [03:41:51] which means I get maybe an hour of time at best with folks in SF before tehy head out [03:41:55] and i'm just waking up at tha tpoint [03:42:50] YuviPanda: ? [03:42:55] It's like 9AM for you now isn't it? [03:42:59] I'm only kind of around [03:43:06] yeah, I figured [03:43:14] Ryan_Lane: anything obvious for '09:06 YuviPanda: Coren: I wonder if 'err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Class[Role::Labsvagrant] is already defined; cannot redefine at /etc/puppet/manifests/role/labsvagrant.pp:5 on node i-000008fe.pmtpa.wmflabs' is a side effect of how that role was added to the interface? '? [03:43:22] bd808: it's 9AM, yeah. [03:43:24] see why it's being defined twice? [03:44:05] Ryan_Lane: nope. git grep tells me it is being defined only once [03:44:38] if you call it from the node, and then it's called somewhere else... [03:44:53] ooohhhhhh [03:44:54] I know [03:44:54] I just added it to the web interface, and ran puppet. [03:45:12] class { 'labsvagrant': -> class { '::labsvagrant': [03:45:21] *facepalm* [03:45:27] otherwise it gets confused about role::labsvagrant [03:45:28] is it trying to recursively include itself now? [03:45:38] it's a stupid puppetism when you aren't using modules [03:45:52] hurr durr [03:47:18] YuviPanda: It's ~21:45 for me; ~09:15 for you. That's a hard time shift to collaborate across :( [03:47:49] bd808: with people keeping normal hours. My usual timezone more closely mastches EST than IST [03:48:24] I forget that you are a young night owl [03:48:26] bd808: ryan merged my patch before leaving [03:48:33] bd808: so running puppetd works now! [03:48:51] * bd808 tries [03:49:01] bd808: run all commands with sudo, btw. [03:49:35] meaning `sudo vagrantlabs ….`? [03:49:56] sudo labsvagrant yaeh [03:53:12] Seems to be working. Of course I enabled the role that takes a really long time to install [03:53:49] hehe [03:53:52] which one? [03:53:57] role::math [03:54:08] heh [03:54:26] It installs a ton of debs and then compiles ocaml source [03:54:56] sounds like fun;) [03:55:45] It seemed pretty easy to script to me but apparently that's because I'm weird. Lots of folks were impressed that I got it to work [03:56:00] or they were just being nice :) [03:56:07] hehe :P [03:56:53] How does the http proxy part work? What's the url to my `multimedia-bd808` instance [03:57:05] right now [03:57:06] you can use [03:57:12] multimedia-bd808.instance-proxy.wmflabs.org [03:57:23] no ssl but [03:57:53] you were working on ssl for that weren't you? [03:58:14] bd808: yeah, so there's no self-serve mechanism for that yet [03:58:22] you tell me instance name + hostname, and I add it for you D [03:58:23] :D [03:58:36] that's how blue-dragons.wmflabs.org is pointing to multimedia-dragons [04:00:07] Man the vagrant-puppet run is slooooow. [04:00:37] yeah [04:00:42] bd808: what is it doing now? [04:01:01] Scheduling refresh of Service[apache2] [04:01:08] right [04:01:17] i'm trying to enable VE and uw [04:02:03] I should have run with verbose so I wouldn't be so bored watching the console [04:02:16] heh [04:02:40] Ah. it was the giant apt-get pull [04:03:59] And it found a bug in the vagrant puppet stuff! [04:04:35] hmm? [04:04:40] the permission issue with git? [04:05:34] no, apache config parts not happening with the right dependency chain [04:05:50] mod-rewrite not loaded soon enough [04:05:56] ah [04:09:10] Yup. second run fixes it. [04:10:03] So we need some other apache related changes for labs usage... [04:10:22] hitting via proxy redirects to localhost with current config [04:10:41] bd808: indeed. I'm 'fixing' it by adding '$wgServer = "http://pearl-hammer.instance-proxy.wmflabs.org";' [04:10:45] to /vagrant/LocalSettings.php [04:10:59] bd808: i've no idea how to fix that by 'default' tho [04:11:24] I'll think about it. It shouldn't be too hard [04:12:23] If noting else we could add a mediawiki-vagrant role that does that for you [04:12:29] *nothing [04:12:50] bd808: that's not the problem, we can do modifications from the labsvagrant module in ops/puppet [04:12:58] bd808: problem is, we don't know the URL it is going to be served over [04:13:22] we could make that configurable from the wikitech interface, tho [04:13:53] ? $(hostname).instance-proxy.wmflabs.org ? [04:14:11] bd808: that's just for testing, really [04:14:25] ah. [04:14:31] bd808: ideally we want something.wmflabs.org [04:15:01] bd808: I don't like the fact that I've to explicitly mention this for mediawiki :( shouldn't we able to do redirects relatively? [04:17:32] mediawiki/LocalSettings.php seems to be the culprit [04:18:08] bd808: hmm? is it setting 127.0.0.1? [04:18:10] * YuviPanda cehcks [04:18:16] yup [04:18:22] oh dear [04:19:06] bd808: so this can be fixed by just getting rid of that line? :D [04:19:07] "When $wgServer is not set the default value is calculated automatically." o_O [04:19:17] https://www.mediawiki.org/wiki/Manual:$wgServer [04:20:34] Yeah, when I comment that line out it works as expected [04:20:50] looks for $varNames = array( 'HTTP_HOST', 'SERVER_NAME', 'HOSTNAME', 'SERVER_ADDR' ); [04:20:54]