[00:00:03] OK! next question… addHost() in OpenStackNovaHost.php. As we discussed, there's a race where the instance may or may not have an IP by the time that gets called, and we check for it. [00:00:27] If the instance doesn't have an IP at that point, will its host entry /ever/ have an ARecord? And does it matter? [00:01:15] (I temporarily thought I caused a regression because my new host records don't have arecs. But… now I'm thinking it's not really a regression.) [00:17:25] cscott: hey [00:17:29] cscott: yes, just woke up [00:17:38] cscott: jet lag has put me in an incredibly reasonable sleep cycle [00:18:52] wondering if i could get you to add bluelake redirecting to port 8001 of towtruck? i'd like to try out your websocket support. [00:19:05] cscott: sure [00:19:10] moment [00:19:17] but it can wait until you wake up ;) [00:19:25] cscott: oh trust me i've woken up [00:19:40] cscott: i was trying to sleep some more, so I can wake up at a more reasonable hour than 5am. gave up [00:20:10] andrewbogott: Ryan_Lane any update on the DNS situation? [00:24:33] cscott: hmm [00:24:34] curl http://towtruck.pmtpa.wmflabs:8001 [00:24:38] curl: (7) couldn't connect to host [00:24:44] am I doing something mindbogglingly stupid? [00:24:58] oh, i haven't started up the server on port 8001 yet. let me do that. [00:25:20] hah! [00:27:20] cscott: poke when started [00:28:52] can you make it port 8080? turns out that's the default. [00:29:10] sure [00:29:34] it's a websocket server, though, so curl doesn't do much. [00:30:16] cscott: sure, i was just using it as a quick and dirty way of testing [00:30:35] cscott: try http://greenlake.wmflabs.org/ [00:30:38] (or https) [00:30:39] curl -v does provide enough output to indicate that there's a server there, though. [00:30:57] too lazy to remove the bluelake mapping :P [00:31:27] this naming convention has a lot of room for expansion [00:31:32] indeed [00:33:30] XMLHttpRequest cannot load https://greenlake.wmflabs.org/findroom?prefix=togetherjstodo&max=5. Origin http://localhost:8888 is not allowed by Access-Control-Allow-Origin [00:33:37] i think that's my problem, though. hang, on. [00:34:19] cscott: yeah [00:34:39] cscott: you can quickly work around it in a temp. way by starting chrome with --disable-web-security [00:34:46] (don't browse the internet with that off tho :P) [00:34:54] cscott: turns off cross domain checking [00:35:00] the server should be emitting the proper cors header, though. [00:37:52] hm, the request seems not to be making it to pinklake. i wonder if your proxy is intercepting it? [00:37:59] pinklake? [00:38:05] cscott: ah, hmm. that's possible. [00:38:10] cscott: wait, 'request' or 'header'? [00:38:40] well, the first request is an OPTIONS method, to check the CORS header. but that request isn't being seem on pinklake. [00:38:47] let me check proxy logs [00:39:15] oh, greenlake is giving me parsoid [00:39:20] that should be port 8000 [00:39:28] so i think it's just pointing at the wrong port. [00:39:38] you asked me to point greenlake at 8000, right? [00:39:43] 8080 [00:39:45] ho [00:39:47] oh [00:39:47] gah [00:39:49] moment [00:39:49] close! [00:39:58] i apparently wasn't fully woken up :P [00:41:22] cscott: http://greenlake.wmflabs.org/ [00:41:29] is that what it is supposed to be? [00:42:22] that looks right! [00:42:28] I'm tailing the logs [00:42:29] ssl cert and all. [00:42:34] yeah :D [00:43:09] let me try to actually hook that up to pinklake now. [00:43:27] :) [00:50:15] YuviPanda: https://wikitech.wikimedia.org/wiki/Projects/mediawiki_Labs_project [00:50:24] * YuviPanda clicks [00:50:39] feel free to change that however you'd like ;) [00:50:45] sweet! [00:50:55] Ryan_Lane: did you look at that gerrit patchset? [00:51:02] also, please name your gerrit stream watcher so that I can reference its name :) [00:51:02] Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/85814/ [00:51:11] Ryan_Lane: it's called gerrit-to-redis [00:51:16] very craetive name... [00:52:50] Ryan_Lane: i should also perhaps document it somewhere... :P [00:53:36] !ping [00:53:37] !pong [00:53:41] -_- [00:54:02] Ryan_Lane: network issues? are you still on a plane? :P [00:55:30] labs-morebots: ... [00:55:30] I am a logbot running on tools-exec-06. [00:55:30] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [00:55:30] To log a message, type !log . [00:55:38] heh [00:55:45] stupid laggy connection [00:55:57] :P [00:57:22] andrewbogott_afk: they will eventually get aRecords, yes [00:57:23] * YuviPanda pokes Ryan_Lane again with https://gerrit.wikimedia.org/r/#/c/85814/ [00:57:28] the MediaWiki job handles that [00:57:36] YuviPanda: did you see my earlier messages? [00:57:39] Ryan_Lane: no? [00:57:45] YuviPanda: https://wikitech.wikimedia.org/wiki/Projects/mediawiki_Labs_project [00:57:49] feel free to change that however you'd like ;) [00:57:50] Ryan_Lane: oh that, yeah I saw that [00:57:54] also, please name your gerrit stream watcher so that I can reference its name :) [00:57:57] Ryan_Lane: nothing about that gerrit change tho :P [00:58:06] Ryan_Lane: i responded to that. it is called 'gerrit-to-redis' :P [00:58:09] very creative name [00:58:14] heh [00:58:15] i should probably document it tho [00:58:52] it just does a publish to redis, or does it stick it into a queue? [00:59:01] Ryan_Lane: sticks it into a queue [00:59:05] ah, ok [00:59:05] Ryan_Lane: the queue is limited to 1024 items [00:59:16] but that's configurable [00:59:35] so, we'd need another process that would pop items from the queue to do actions [00:59:47] I'll update the doc soonish [00:59:56] indeed [00:59:59] I can make a python daemon to do that [01:00:00] easy enough [01:00:04] yup [01:00:15] it could actually be sartoris [01:00:15] Ryan_Lane: BUT ALSO MERGE MY CURRENT PATCH! :P [01:00:28] Ryan_Lane: is there docs on git-deploy? [01:00:29] this just adds a script? [01:00:37] only kind of [01:00:40] I need to update those docs [01:00:41] Ryan_Lane: the patch? it adds a module too. [01:01:09] mode => 'a=rx,o=rwx', ??? [01:01:13] dude, use numbers ;) [01:01:20] 755 [01:01:34] 0755, to be more exact [01:01:36] Ryan_Lane: pffft [01:01:38] Ryan_Lane: whhhyyy [01:01:43] Ryan_Lane: i like this better. [01:01:45] if you do that, I'll merge it [01:01:50] also, fuck, I don't think I meant o [01:02:16] jesus, I was using that to make sure I don't have to use numbers, and i get that wrong too. [01:02:17] grr [01:02:19] Ryan_Lane: okay, fixing [01:03:20] !ping [01:03:20] !pong [01:03:22] Ryan_Lane: pushed. [01:03:28] labs-morebots: ... [01:03:28] I am a logbot running on tools-exec-06. [01:03:28] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [01:03:28] To log a message, type !log . [01:04:17] Ryan_Lane: updated patchset. [01:05:05] one sec [01:05:39] if you're lucky I'll be able to ssh into production [01:05:39] * Ryan_Lane is on a plane [01:05:55] Ryan_Lane: hehe :P [01:05:58] Ryan_Lane: still on a plane? [01:06:09] yep [01:06:15] for another couple hours ;) [01:06:15] Ryan_Lane: also, https://gerrit.wikimedia.org/r/#/c/84926/, do you agree with Coren's suggestion that *all* labs related roles should be on a single labs.pp? [01:06:28] um [01:06:31] Ryan_Lane: instead of being as they are now, with a file per related role-grouping? [01:06:34] I dunno [01:06:41] I prefer the file per related role-grouping thing [01:06:51] if this was in a module, it would be a file per [01:07:44] indeed [01:07:53] so file per seems to be the 'right thing' for me [01:08:01] one for toollabs, one for bastions, etc. [01:08:07] which is what that change does :P [01:09:09] oh. ugh [01:09:14] you may not want to use /vagrant [01:09:18] why? [01:09:25] because that is in / [01:09:28] and / is small [01:09:30] file space issues? [01:09:35] yes [01:09:37] this is only configuration.... [01:09:46] it's not mediawiki too? [01:09:52] well, configuration and code :P [01:09:58] right [01:10:02] so, it could get massive [01:10:02] I should probably move it to /mnt [01:10:05] yeah [01:10:37] Ryan_Lane: but can you merge it now anyway? [01:10:47] Ryan_Lane: the vagrant codebase hard codes /vagrant a couple of places [01:10:57] Ryan_Lane: so I'll need to refactor that. or use a symlink [01:11:02] i'd rather refactor it properly. [01:11:06] well, you could link /vagrant [01:11:19] /vagrant -> /mnt [01:11:27] yeah, but hardcoding is bad! [01:11:32] so I want to fix it there. [01:11:34] or even better /vagrant /mnt/vagrant [01:11:41] yeah, but before people start using this.... [01:11:49] i've not added a role, so nobody can use it yet :P [01:12:02] fine, let me do that [01:12:05] give me a minute [01:12:11] or just make it use /mnt/vagrant [01:12:23] and then refactor the other stuff :) [01:12:37] :P [01:12:59] someone *will* screw themselves otherwise ;) [01:14:40] hooray! https://launchpad.net/manila [01:15:13] maybe we'll finally be able to get rid of those damn scripts that manage NFS/gluster [01:15:36] Ryan_Lane: updated patchset, testing now. [01:15:45] YuviPanda: ok, https://pinklake.wmflabs.org is now using greenlake and we have no more dependencies on external servers. whoo. [01:15:52] YuviPanda: your websocket support seems to be solid. [01:15:52] wooohoo! [01:15:56] \o/ [01:16:11] nailing https *and* websockets! [01:16:26] although chrome doesn't seem to like protocol-relative URLs for websockets, so we're https-only on the collab part. [01:17:02] i could probably hack around that with js to add back in an explicit protocol, but it's not worth it at this point [01:17:19] it seems like that's a good bug to open with google [01:17:57] well, webkit and blink(?) [01:18:27] Ryan_Lane: seems to work :D [01:18:50] cscott: does that make things any faster? [01:18:56] cscott: you should also consider making a vagrant role for this ;) [01:19:26] YuviPanda: vagrant? or puppet? ;-) [01:19:33] cscott: puppet in vagrant :P [01:19:41] cscott: not operations/puppet.git, but mediawiki/vagrant.git [01:21:05] Ryan_Lane: we need to settle the 'one file for each set of labs roles' or 'one file for all labs roles' question yet [01:21:13] Ryan_Lane: should settle before I add the role [01:21:37] well, for now it can all go into one file [01:21:52] when roles go into a module, it'll need to get split up [01:21:56] Ryan_Lane: well, it *already* is in seven different files. [01:22:11] Ryan_Lane: ls labs*.pp | wc -l is 7 right now [01:22:14] heh [01:22:22] and labs.pp itself has only toollabs things [01:22:32] * Ryan_Lane rolls his eyes [01:22:42] Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/84926/ fixes that :P [01:22:43] I'd say different files [01:22:49] Ryan_Lane: can you merge? :D [01:22:54] +1 to different files [01:22:55] no chance I'm merging that right now [01:22:59] pfft [01:23:03] fine, let me do a lower risk one [01:23:18] heh [01:23:32] Ryan_Lane: where are you flying to? [01:24:49] NYC [01:24:57] gonna be there for a couple weeks [01:25:24] Ryan_Lane: oooh, nice! [01:25:46] Ryan_Lane: https://gerrit.wikimedia.org/r/85946 is less controversial, at this point :P [01:29:05] hurr durr [01:29:05] he left [01:29:22] legoktm: the vagrant / labs patch was merged :P [01:29:56] woot [01:30:09] legoktm: needs a little more work, but is almost there :D [01:30:31] woo, ty, Ryan_Lane. [01:30:35] yw [01:31:56] now to add the final part, enable-disable role [01:32:03] * YuviPanda considers reusing the ruby cod that already exists [01:38:11] Ryan_Lane: also +2 another trivial https://gerrit.wikimedia.org/r/#/c/84927/? [01:42:33] YuviPanda: what other types of cod are there? [01:42:46] cscott: cod? [01:43:01] you were talking about the existing ruby cod [01:43:09] which sounds tasty [01:43:14] gah [01:43:15] :D [01:43:25] 07:08 YuviPanda: Ryan_Lane: also +2 another trivial https://gerrit.wikimedia.org/r/#/c/84927/? [01:43:30] (in case you missed it the first time :P) [01:44:53] YuviPanda: that's not how you're supposed to use the system_role description [01:45:02] the description is shown to users when they log in [01:45:23] hmm, in that case I guess there doesn't even need to be one? [01:45:26] so, something like "Labs bastion host" would be a better description [01:45:27] since there's an MOTD anyway [01:45:28] ok [01:46:28] Ryan_Lane: updated, pushing. [01:46:30] pushed [01:46:38] ok [01:48:36] Ryan_Lane: i'll wait for someone not on a plane to apply that role to labs bastions, since if that goes wrong it'll prevent access to labs bastions :P [01:48:57] heh, yeah [01:51:02] legoktm: so the update enabling mosh for labs bastions just got merged. needs someone to apply it, should be able to find someone later today :) [01:51:08] omg [01:51:11] yessssss [01:51:29] legoktm: also, dammit, I keep thinking 'soon the rest of the US will wake up!' while in reality everyone will go to sleep and i'll be all alone on IRC [01:51:30] grrr [01:51:34] * YuviPanda kicks Jetlag [13:53:16] Coren: around? [13:53:27] Yep. What's up? [13:54:01] Coren: can you add role::labs::bastion to the labs bastions? [13:54:18] Coren: Ryan_Lane merged them, but he was on a plane so couldn't do add the roles [13:54:28] Yeah, simply enough. Give me a minute. [13:54:47] Coren: woo, sweet [14:00:17] Wait, you made it a role? I need to add this to the wikitech interface then. [14:00:50] Coren: there has been no bastion role at all [14:00:52] Coren: ever [14:00:55] Coren: so I made a role [14:01:17] Coren: toollabs has its bastion role, but the bastion project has no roles. [14:04:03] I've added it to the global puppet classes, in the "roles" group. [14:04:29] So it can be added anywhere with just the checkmark. [14:04:43] You want me to actually add it to the labs bastions only? [14:05:16] Coren: yeah [14:05:25] Coren: because I can't add them :P [14:05:55] Well, you could /now/. :-) I'm there already though. [14:07:27] Coren: to be clearer, we're talking about the machines in the 'bastion' project, right? :P [14:07:30] not in toollabs [14:07:49] Ah, no. I got confused and just applied this to tool labs. [14:07:51] bastion1, bastion2, bastion3 [14:07:52] haha [14:08:13] in my defence, I said labs! :P [14:13:02] "interesting". E: Couldn't find package mosh [14:13:49] Ah. Mosh is in universe. [14:14:15] where do we add universe to the sources? [14:14:17] or do we, at all? [14:25:35] Coren: any luck? [14:33:06] Coren: also, see https://wikitech.wikimedia.org/wiki/Projects/mediawiki_Labs_project - beginnings of the MWLabs project :) [14:35:50] hey nerus1 [14:36:13] * YuviPanda pokes Coren again, slowly [14:45:18] YuviPanda: Sorry, got distracted. I'm trying to find the "right" way to add universe; the new images do it by default. [14:45:27] aaah [14:45:30] copy paste, perhaps? ;) [14:45:51] Coren: we could maybe even just add another bastion... [14:50:31] Odd. Universe should normally be added by modules/apt/manifests/init.pp [14:51:00] Ah, no, that's "our" universe. [14:51:11] Still should work. Hm. [14:51:43] o_O [14:51:52] random oddities? [14:53:01] Coren: are all the views that were lost in the DB outage now recreated? Ref bug 54390. If so, I'll mark it as fixed. [14:53:22] As far as I can tell, universe /is/ included but I can't find the package with apt. [14:53:41] update maybe? [14:53:51] Nettrom: The process that adds them is complete, but I haven't checked against those reported in the bug. If you see them there, feel free to close. [14:54:17] YuviPanda: Worth a try, but I doubt mosh is a recent addition. :-) [14:54:26] Oh! [14:54:29] lucid [14:54:36] Coren: the only one I knew about was arwikiquote_p, and that one's working now... I'll go ahead and mark it fixed then, someone else can reopen if we're wrong [14:54:45] bastions are running lucid and not precise. [14:55:04] YuviPanda: hey [14:55:11] wat [14:55:23] Coren: > ubuntu-12.04-precise (deprecated) [14:55:30] Ah, bastion-restricted may be alone in that. [14:55:32] Coren: bastion 1, 2, 3 run precise, apparently [14:55:35] [bz] (8RESOLVED - created by: 2Morten Wang, priority: 4Unprioritized - 6major) [Bug 54390] Re-create database views lost in the outage such as arwikiquote_p - https://bugzilla.wikimedia.org/show_bug.cgi?id=54390 [14:55:35] indeed [14:55:38] let's ignore that one :P [14:55:42] Which is where I was testing. [14:56:04] Can't ignore it; the inclusion of mosh breaks puppet run. You need to add a check for lucid in your class. [14:56:22] ah [14:56:25] waaat [14:56:34] Coren: why do we even have -restricted? [14:57:28] opsen are not allowed to log in the others; and vice-versa. Necessary to guard against accidental or necessary key forwarding. [14:58:04] But, arguably, -restricted should really be upgraded to precise. [14:59:10] indeed [14:59:24] i don't even know if we have any other lucid ones [15:01:56] Just started the ball rolling on that topic. [15:02:08] Coren: it breaks puppet on that host only? [15:02:16] Coren: you can just not apply it to that one :P [15:03:36] YuviPanda: Yeah, that's what I'll do in the meantime. [15:03:40] sweet [15:03:46] no mosh for opsen! [15:06:13] Real opsen use telnet with a hardcopy teletype. :-) [15:06:44] At least they did, 45 years ago. :-P [15:07:36] !excuse is here's why it's broken: telnet towerl.blinkenlights.nl 666 [15:07:36] Key was added [15:07:47] !del excuse [15:07:47] If you want to remove a key, type !excuse del [15:07:49] Coren: boooorrrring :P [15:07:52] !excuse del [15:07:53] Successfully removed excuse [15:08:04] !excuse is here's why it's broken: telnet towel.blinkenlights.nl 666 [15:08:05] Key was added [15:47:59] Coren: https://gerrit.wikimedia.org/r/#/c/84969/ [15:48:41] (03CR) 10coren: [C: 032] "Yeay uri changes!" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/84969 (owner: 10Krinkle) [15:49:02] (03CR) 10coren: [V: 032] "Yeay uri changes!" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/84969 (owner: 10Krinkle) [15:50:08] Krinkle: Pushed and updated. [15:51:44] Thx [16:09:12] YuviPanda, tell me about MWLabs? [17:04:23] andrewbogott: https://wikitech.wikimedia.org/wiki/Projects/mediawiki_Labs_project is all we have now [17:04:34] andrewbogott: however, the mediawiki_singlenode replacement is labsvagrant, and that has been merged! [17:04:51] needs one more patch for a role enabling, disabling script and then it can be documented and publicized [17:05:10] huh, cool. [17:05:33] btw, mediawiki_singlenode, there's still that pending patch about changing the permissions and i'm +/-0 [17:05:35] andrewbogott: it is the labsvagrant module, check it out! it uses the mediawiki/vagrant.git puppet stuff, without using any of the actual vagrant stuff [17:05:50] waves bye though, cya later [17:05:52] mutante: andrewbogott I'd ideally like to get rid of mediawiki_singlenode at some point. [17:07:09] !ping [17:07:10] !pong [17:07:10] gah [17:07:55] !ping [17:07:55] !pong [17:08:42] !ping [17:08:42] !pong [17:08:45] :D [17:10:38] Ryan_Lane: i'm on a train [17:10:46] Ryan_Lane: so not the best of internets [17:10:56] Ryan_Lane: also, Coren found that one of the labs bastions is still on lucid! [17:11:01] Coren: did you apply the role to the other ones? [17:13:15] heh. lucid, eh? [17:13:17] not surprising [17:13:37] well, we can create a new instance, move the ssh keys, then move the floating IP [17:13:43] that's the whole point of floating ips [17:14:19] Ryan_Lane: it is the restricted bastion [17:35:04] Ryan_Lane, the DNS code is ready for you to look at again, whenever. I found the bug I was hunting yesterday. [17:35:21] ah, cool [17:35:26] I'll look at that now [17:47:06] andrewbogott: you're using the wrong brace style for MW ;) [17:47:44] Oh, did I put my 'else' on its own line? [17:47:50] I tried to go through and fix those but I must've missed one [17:48:01] Anyway, mark 'em and I'll fix. [17:48:08] I mean the one true brace style [17:48:14] MW uses inline brace [17:48:28] Oh… that'll be a harder thing for me to learn :( [17:48:33] :) [17:48:35] Want me to fix now, or should I wait for you to finish reading? [17:48:38] * Ryan_Lane prefers inline brace [17:48:42] can fix now [17:48:48] well [17:48:50] maybe wait [17:48:55] because I may have other feedback [17:48:58] ok [17:51:45] so, this one looks good, once the brace style is fixed: https://gerrit.wikimedia.org/r/#/c/85991/1 [18:02:37] andrewbogott: think you can take a stab at the proxy interface GUI code, once the DNS stuff is fixed? [18:02:53] YuviPanda|train: dns code was step 1 of that ;) [18:02:59] figured :D [18:03:07] YuviPanda|train: probably not today but, yes, soon [18:03:12] of course [18:03:18] still, yay [18:17:25] andrewbogott: reviewed [18:17:35] thanks! Will amend. [18:17:37] other than some minor issues it looks fine [18:17:45] good work :) [18:24:33] Oops, missed something... [18:28:19] ok, I think that's everything. [18:37:14] ok. I'll need to modify all the entries, too [18:37:31] yeah, I'm looking forward to not doing that :) [18:37:46] hm. maybe I should wait till tomorrow morning to do this, so that I have a full day to handle any breakages :) [18:37:54]