[03:16:33] hi, guys, how to install php on an instance? I tried configure->webserver::php5 but it does n't work. [03:48:46] hey jeremyb, you there? [03:49:25] * jeremyb is [03:49:32] kinda [03:49:56] can i take you up on your offer? i'd like to set up phabricator on labs [03:50:01] could you create a project for it? [03:50:16] i has no such powers [03:50:27] although i think such a project exists already [03:51:09] it does so in fact [03:51:15] !resource phabricator | ori-l [03:51:16] ori-l: https://labsconsole.wikimedia.org/wiki/Nova_Resource:phabricator [03:51:29] 20:47 (Eloquence) marktraceur, yeah johnduhart had it set up for testing a few months ago, but it may be easier to set it up from scratch at this point [19:32] [03:51:45] you should use the same project i think though [03:51:53] could you add me to the project? or is that something only ryan can do? [03:51:56] unless you want a semi-prod one and a staging one [03:52:44] ori-l: any current member can give you the same rights they already have (ryan/mutante/johnduhart). any wiki sysop can give you anything they want [03:52:57] ori-l: of course ryan and mutante are also sysops ;) [03:53:18] * ori-l winks salaciously at mutante [03:53:30] haha [03:54:00] ori-l: i have this periodic need for a way to query TZ from people ;) [03:54:09] currently i'm thinking mutante [03:54:27] i can't remember who's moved to SF or not (i know someone did recently) [03:54:28] i think there's a convention -- something like /ctcp mutante time [03:54:36] that doesn't do anything i think [03:54:50] my client uses UTC and i certainly don't live in UTC [03:54:56] or maybe i'm rare [03:55:36] hi, guys, how to install php on an instance? I tried configure->webserver::php5 but it does n't work. [03:55:47] MichaelShavlovsk: maybe tell us more? [03:56:05] what's your test for doesn't work? what's happening with puppet? are runs succeeding? [03:56:07] "http://www.irchelp.org/irchelp/rfc/ctcpspec.html" [03:56:29] "TIME - Gets the local date and time from other clients. " [03:56:36] so according to the specs, at least, it ought to be local [03:56:42] ori-l: it says CEST for mutante i think [03:56:45] i think lots of clients block it because ctcp was an attack vector [03:56:46] ori-l: right, it *is* local [03:57:28] ori-l: the local time configured on the box where my irssi is running is UTC. the box is a virtual server in new jersey (US) and I am in brooklyn (NY, US) [03:57:48] oh! where in brooklyn? [03:58:05] CTCP HOOD [03:58:21] * ori-l is an ex-new-yoker, miss the city terribly [03:59:43] ori-l: park slope. otto's relatively close too [04:00:12] jeremyb: after I created an instance I go to configur then I choose among other options webserver:php5, then I logging into the instance and on "php -version" it says that php is not installed. Is one suppose to install php though these puppets in configure of an instance? [04:00:54] MichaelShavlovsk: webserver::php5 doesn't mean php5-cli to me (just thinking about it) [04:01:13] a server need not have php on the path to be said to have php installed [04:01:51] MichaelShavlovsk: `dpkg -l 'php*'` says what? [04:02:01] jeremyb: cool [04:02:35] do you have any experience with puppet, btw? [04:03:32] me? no [04:04:00] sorry, that was at jeremyb [04:04:00] jeremyb: yes there is php-common [04:04:09] ori-l: some [04:04:18] MichaelShavlovsk: and php5-cli is what? [04:04:23] uninstalled? [04:04:42] jeremyb: i'm new to it, so i'm wondering which already-puppetized piece of software might be closest to phabricator so i could work off of its manifest [04:04:47] ori-l: no haifa for you right? [04:04:54] oooh [04:04:56] jeremyb: nope [04:05:03] well gerrit is the closest function obviously [04:05:05] jeremyb: there isn't line with php5-cli [04:05:07] but it's a php app [04:05:10] not java [04:06:13] MichaelShavlovsk: so that's your problem [04:06:52] MichaelShavlovsk: and it's perfectly reasonable (IMO as I just said) that webserver::php5 would not install php5-cli [04:07:27] jeremyb: okay, thank you. [04:08:22] MichaelShavlovsk: maybe you want apaches::packages or maybe you need to change a manifest. or just install without puppet [04:09:04] jeremyb: but it does not allow me to install packages [04:09:36] jeremyb: is there way to install packages using apt-get? [04:10:46] MichaelShavlovsk: huh? who doesn't? [04:13:35] jeremyb: in the server I got message that something like "you are not in sudo errs", then I went to "mange sudo policies" and created policy, after that I had " mshavlovsky is not allowed to run sudo on ..." [04:13:54] nah, don't deal with manage sudo policies [04:14:11] just make sure you're a sysadmin for whichever nova project that box is in [04:14:14] which box [04:14:16] ? [04:14:50] box? [04:16:24] do you mean an instance? it is i-0000039e.pmtpa.wmflabs [04:18:10] jeremyb: how can I add myself in sysadmin? I am in netadmin now [04:18:30] yes, instance [04:18:39] netadmin doesn't matter here [04:18:47] i don't think you can add yourself [04:19:17] i can't even see whether or not you're a sysadmin i think [04:19:36] okay, so I can use only avaliable puppets [04:20:57] it's hard to imagine you could change puppet without being sysadmin [04:20:57] that's a bug if you can IMO [04:20:57] I cannot [04:21:09] erm? [04:22:03] I meant I can use only avaliable puppets through configuration, right? [04:24:44] i think you don't know what puppet is maybe? [04:24:52] can you rephrase without saying puppet? [04:32:16] jeremyb:To install packages I can do next: in instance list I click configure on my instance. Then I see, "puppet information" and "global groups" and lots of check-boxes like "mysql::config", so I can check whatever configuration I needed and click submit. Is this the only way to get packages on the instance? Even though I have mysql-common package I cannot browse databases and I need it. [04:36:36] jeremyb: if I have mysql-common package does it mean that I have installed mysql database? [04:37:01] no [04:43:19] jeremyb: so I am trying to configure the instance for installation mediawiki, Do you know how can I install necessary parts like mysql and php on the instance? I found that I can like written here https://labsconsole.wikimedia.org/wiki/Help:Instances#Configuring_instance , but it does not work, I checked all thing with mysql and I have only mysql-common package instance. Whether configuration does not work or there is [04:43:19] other way to install mysql. [04:44:29] checking everything with mysql is likely to cause problems not solve them [04:46:29] okay, I did not find explanation what every part means but here they are : db::core; db::es::master; db::es:slave; mysql::conf; mysql::datadirs; mysql::mysqluser; role::labs-mysql-server [04:48:17] jeremyb: what of the about do I need, do you know? [04:48:32] i haven't a clue [04:49:10] there has been talk of making a class that just installs a complete+working mediawiki instance. afaik it's not been created yet [04:50:02] jeremyb: okay, thank you [04:53:01] MichaelShavlovsk: you're sure you're not a sysadmin right? checked [[special:novaproject]] ? [04:55:58] jeremyb: yep, it looks like I am sysadmin, but why I cannot add packages? do you know? [04:56:18] MichaelShavlovsk: what did you try to add them? [04:56:23] i think you still haven't told me [04:58:10] jeremyb: sorry, may be I undertood you, I want just install mediawiki on the instance, so I need apache, php and myslq, I cannot do it through sudo apt-get install. [04:58:26] MichaelShavlovsk: ohhh, sudo again [04:58:27] sorry [05:00:38] MichaelShavlovsk: so, what's the message exactly? can you try just `sudo echo foo` for me? [05:00:50] MichaelShavlovsk: what password do you give it when it prompts you? [05:00:59] should be the one you log in to labsconsole with [05:01:14] jeremyb: mshavlovsky is not allowed to run sudo on i-0000039e. This incident will be reported. [05:01:25] for sudo echo foo ? [05:01:32] yep [05:02:19] did you log out and back in to the box recently? [05:02:26] were you made sysadmin recently? [05:03:11] I did log out and back in to the box recently [05:03:57] ohhh, remove the sudo policy btw [05:03:57] what do you mean where I made sysadmin recently, I am listed as sysadmin in labsconsole.wikimedia.org/wiki/Special:NovaProject [05:04:01] okay [05:04:16] you're probably stopping yourself from echoing [05:04:55] it's not sysadmin that gives you sudo [05:05:02] it's the sudo policy that does [05:05:11] Ryan_Lane: but there's a default sudo policy, right? [05:05:13] no [05:05:22] Ryan_Lane: remember how i complained about not being able to see who's a sysadmin/netadmin for other people's projects? i found the right ldap incantations ;) [05:05:26] heh [05:05:29] huh [05:05:44] it might be a good idea to generate a default sudo policy [05:05:49] i guess there's just a reallllllly common sudo policy on labs [05:05:54] I'd need to add that to openstackmanager [05:05:57] !sudo [05:05:57] You have sudo in any project that you are a member of, excluding global projects (like bastion). Your sudo password is your labsconsole wiki password. [05:06:00] hm [05:06:04] @search sudo [05:06:04] Results (Found 1): sudo, [05:06:07] damn it [05:06:15] hah [05:06:22] @apropos sudo [05:06:32] !sudo-policy is https://labsconsole.wikimedia.org/wiki/Help:Sudo_Policies [05:06:32] Key was added [05:06:48] !sudo-policies alias sudo-policy [05:06:48] Created new alias for this key [05:07:16] could probably make that documentation a little better [05:07:23] hey Ryan_Lane, could you add mark traceur and i to the (seemingly abandoned) phabricator project on labs? [05:07:31] sure [05:07:31] we're in the process of puppetizing it [05:07:36] * Ryan_Lane nods [05:07:43] thanks! [05:07:52] obviously an external ip would be useful too :) [05:08:33] OrenBo: usernames? [05:08:35] Ryan_Lane: /me too for phab ? [05:08:49] I'll add one of you [05:08:51] we're collaborating at http://etherpad.wikimedia.org/phab, probably doing all sorts of horrible things since neither of us has any real experience with puppet. if you guys want to take a look, we'd welcome input [05:09:04] i'm "ori" [05:09:05] isn't it premature to puppetize it? :) [05:09:15] your wiki username is ori? [05:09:19] Ryan_Lane: there's a wikitech-l thread [05:09:28] it's Ori.livneh, but gerrit didn't like the dot and truncated the latter half [05:09:37] ori-l: gerrit ? [05:09:55] ugh, well it's Ori.livneh in labs [05:10:01] fucking gerrit [05:10:12] people need to tell me when there are username issues like that [05:10:19] i did, a while ago [05:10:21] I'll change the allowed characters in labsconsole [05:10:50] ori-l: I added you. you can add the others [05:11:03] cool will do -- jeremyb, username? [05:11:30] ori-l: ==nick [05:12:15] ori-l: huh, i think there's no indication that I can find that gerrit doesn't like the dot. (when searching for you or clicking on you etc.) [05:12:53] i don't remember precisely where and how it choked, but try it and you'll see [05:12:57] unless Ryan_Lane just fixed it [05:13:03] no [05:13:12] it's with the dot in labsconsole [05:13:12] no, he was going to just block new bad people from being created [05:13:14] which means in ldap [05:13:47] jeremyb: thank you, now I can do sudo [05:14:11] MichaelShavlovsk: cool [05:14:38] MichaelShavlovsk: i guess i just never encountered a project without that normal sudo policy before. [05:14:54] yeah, I'm usually nice and set it up manually [05:15:02] since it is often a point of confusion [05:15:02] =)) [05:15:17] I think it's more likely people want the default than to set up something custom [05:15:26] I'll just make openstackmanager make it by default [05:16:53] Ryan_Lane: what do you think would be a sensible instance size for phabricator? [05:17:17] i imagine it'll host between 2 and 5 projects during the eval [05:17:25] small [05:17:47] when we upgrade openstack, you'll be able to resize instances ;) [05:18:05] that's awesome [05:18:10] isn't *that* exciting? :) [05:18:11] heh [05:18:24] we'll have snapshot support too, but I don't know if I'm going to add that [05:18:30] I have a feeling that's going to end bad places [05:18:57] in terms of taking up lots of storage space? [05:18:59] !bug 39092 [05:18:59] https://bugzilla.wikimedia.org/39092 [05:19:10] ori-l: yeah, and people being lazy [05:19:23] snapshots require downtime, too [05:19:24] which availability zone is liklier to be stable in the next few weeks? [05:19:30] they are all the same [05:19:37] there's only pmtpa [05:20:06] and it'll have downtime for the upgrade [05:20:18] Ryan_Lane: can't it be more like security groups? there's a set of default policies and you can choose to use one of those or make your own? [05:20:18] that said, I think you can live with a day or so of downtime ;) [05:20:34] jeremyb: you can right now [05:20:34] hey guys, I want create a git repository and I did request at http://www.mediawiki.org/wiki/Git/New_repositories/Requests , but nothing happens since friday, is this the right way to obtain git repository for a project? [05:20:42] jeremyb: that's exactly how it works [05:20:53] MichaelShavlovsk: yeah [05:21:04] ori-l: where will the repos canonical homes be? i was thinking they can live on gerrit? [05:21:07] I'm not sure why it's taking so long [05:21:20] i wonder if there's a way to disable review in gerrit [05:21:28] or just disable push for review i guess [05:21:31] per repo [05:21:37] jeremyb: you can have push rights [05:21:40] you can do straight pushes to the repos [05:21:57] of course, that defeats the entire purpose [05:22:00] but that is illegal and nobody ever does that [05:22:08] Ryan_Lane: right. but some people will do push for review instead. i want to prohibit that [05:22:22] don't involve gerrit in the chain [05:22:33] Ryan_Lane: we can have push to personal sandbox with some convention and have phabricator watch for personal sanxboxes [05:22:36] if you are going to evaluate it, do so in a way we can actually use [05:22:54] hrmmmm [05:23:09] so, repos on same box with phabricator then... [05:23:11] which means no gerrit and no github [05:23:18] gitolite or something maybe [05:23:24] that said [05:23:25] +phabricator [05:23:30] I don't understand why we're looking at phabricator *at all* [05:23:35] heh [05:23:40] it doesn't do repo management, how is it even a replacement for gerrit!? [05:23:42] if you just want to try out the code review parts until it does repo hosting, have at it [05:23:45] paravoid: it's in the plans [05:24:00] same with ACLs and private repos [05:24:30] but at the same time noone gives a shit about gitlab which it does all that already [05:24:43] there's a pretty strong incentive to go with phabricator [05:24:55] yeah, I can see that, that's why I'm wondering. [05:25:08] the phabricator guy came to the office today and showed us the system [05:25:12] what's the incentive? [05:25:18] paravoid: so... c_rehash? [05:25:27] 1. It's in php [05:25:32] jeremyb: in a sec. [05:25:35] which means our devs can actually make changes [05:25:44] 2. it's code review was actually pretty damn nice [05:25:55] 3. it's not pull request based [05:26:15] 4. it supports post-commit review [05:26:25] Ryan_Lane: how is it based then? [05:26:36] gitlabs is basically a github clone. github does code review fairly poorly [05:26:43] jeremyb: eh? [05:26:45] based/ [05:26:47] I like pull request based to be frank :) [05:26:56] I hate pull requests [05:27:06] it would be a coup for evan priestly (chief author / maintainer) to list us as a user, and he seemed eager to implement various requirements we have [05:27:15] indeed [05:27:16] that should also be noted [05:27:28] much better than "have a million of different completely separate reviews" or "squash everything into a single commit" [05:27:29] that said, I think we'll be using gerrit for quite a while [05:27:36] paravoid: it does neither [05:27:55] gerrit fails miserably when you try the normal (in the git world) topic branch workflow [05:28:12] I disagree [05:28:13] not sure what phabricator does [05:28:43] btw if once of you have a sec maybe you can hit "submit" on https://rt.wikimedia.org/Ticket/Display.html?id=3325 / https://gerrit.wikimedia.org/r/#/c/17040/ -- all the approvals are in [05:28:45] when you use git review it abstracts that stuff away from you [05:28:47] *one [05:29:05] how does it abstract the fact that you lose information? [05:29:11] you don't [05:29:13] it's in the change [05:29:14] the feature was implemented in a topic branch, you lose this information forever. [05:29:23] Ryan_Lane: if not pull req based then what takes the place of pull reqs? is there a diagram? ;) [05:29:23] your entire history is in the change [05:29:32] jeremyb: you push a diff in [05:29:43] hrmmmm [05:29:46] afaik in gerrit you merge individual commits, not branches [05:30:00] paravoid: yes, so you end up with a clean master [05:30:08] the change is a full branch [05:30:09] will I see 'merge branch "puppet-module"' in the git history? [05:30:18] you can pull all of the history from it [05:30:19] if that was a local topic branch? [05:30:48] I like that fact that master isn't cluttered with a million mistakes people made [05:30:52] you lose merge info [05:30:57] who cares? [05:31:00] our master is /clean/!?! [05:31:12] it's full of completely useless merges [05:31:20] puppet's master isn't clean because we had to merge to test anything previously [05:31:20] * jeremyb chuckles [05:31:27] that's not gerrit's fault [05:31:37] that's not the reason [05:31:47] look at all of your recent commits [05:31:56] it's basically "commit foo; merge foo" for all of them [05:32:23] and dont' tell it's because you didn't pull before pushing, because I've seen it do that even if I do that... [05:32:38] * Ryan_Lane shrugs [05:32:50] but that's a separate issue from the "is not compatible with topic branches workflow" [05:33:00] we can turn ff-only on in our repo [05:33:07] but then you'd need to rebase more often [05:33:18] seriously, look at the past 10 commits of yours [05:33:28] the question is "is the HEAD at time of merge also the parent of your commit? if so then it's a fast forward and no merge commit. if not then merge commit" [05:33:28] I'm not using git-review [05:33:31] AFAICT [05:33:44] how's git-review relevant? [05:33:50] it rebases for ou [05:33:51] *you [05:34:02] i've not installed git-review myself [05:34:14] I've seen it do merges even if I was commiting on top of the latest head [05:34:24] paravoid: hard to believe [05:34:29] again, we can turn ff-only on, if you'd like [05:34:58] iirc, it was because I did +2 and submit in two different steps or something [05:35:17] ^demon has an open bug on this [05:35:23] oh heh. [05:36:20] so, we get useless merges and we also don't get useful merges, because it treats every branch merge as individual commits [05:36:56] that's my biggest complaint with gerrit. I didn't mind the colors or the pre-upgrade slowness or anything [05:37:08] I never find that to be a problem ;) [05:37:20] there's also workarounds for that in git log [05:37:30] there's threads about this on wikitech-l [05:38:07] what are your arguments against pull requests? [05:38:22] I've never found it to be a problem and I wonder [05:38:32] my problem is with forking [05:38:56] you mean the same software all over github under different usernames? [05:39:02] yes [05:39:03] oh I hate that too [05:39:07] totally hate it [05:39:10] which is a requirement of pull requests [05:39:23] is it? I haven't seen how gitlab does it [05:39:31] it doesn't /have/ to be [05:39:33] pull requests also put the burden of merging on the reviewer [05:39:41] ^ that part sucks [05:39:49] what do you mean? [05:40:08] you make a pull request. the reviewer has to actually merge it [05:40:25] oh you mean merge conflicts [05:40:27] yes [05:40:30] and that sucks [05:40:50] well, again, it doesn't have to be that way [05:41:06] you can say "doesn't merge cleanly, fix and resubmit the merge" [05:41:42] same with commits that conflict as we have it now [05:41:45] some labs instance is misbehaving [05:43:19] dumps-inc [05:43:26] where's hydriz? [05:43:26] heh [05:43:45] gitlab's graph thing is really cool [05:43:50] Ryan_Lane: it's that daily? [05:44:00] and it really shows how a nice git workflow would work [05:44:01] paravoid: like github's network? [05:44:10] yeah [05:44:19] jeremyb: more often than that http://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Virtualization+cluster+pmtpa&h=virt8.pmtpa.wmnet&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [05:44:23] gitlab's basically a github ripoff, so... [05:44:45] I don't think github's model would work that well for us, honestly [05:45:04] Ryan_Lane: you mean if it were released DFSG free tomorrow? [05:45:08] yes [05:45:20] it's funny how github is a popular option [05:45:41] I'm betting it isn't very popular with orgs that have over a hundred repos [05:45:44] I should probably put work where my mouth is and set up a gitlab instance [05:46:01] I'll even do curl | sh! [05:46:15] * Ryan_Lane stabs [05:46:31] which wgets ruby and installs it [05:46:36] -_- [05:46:36] from source [05:46:42] * Ryan_Lane groans [05:46:44] paravoid: hydroxide and I will be at this talk thursday night. i'll see how fast we can get the video. http://nylug.org/home/index.shtml [05:46:52] and this is why I hate every fucking ruby app in existence [05:47:51] apt-get install curl sudo [05:47:51] # 3 steps in 1 command :) [05:47:52] curl https://raw.github.com/gitlabhq/gitlabhq/master/doc/debian_ubuntu.sh | sh [05:47:55] :-) [05:48:03] that's what it says I should do! [05:48:25] wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p194.tar.gz [05:48:25] tar xfvz ruby-1.9.3-p194.tar.gz [05:48:26] cd ruby-1.9.3-p194 [05:48:26] ./configure [05:48:26] make [05:48:28] sudo make install [05:48:31] paravoid: not `curl | sudo sh` ?? [05:48:37] make, make install. sigh [05:48:46] i guess maybe it prompts for password later [05:48:49] what year is it again? [05:48:51] jeremyb: nah, it does sudo multiple times in the script [05:49:16] this is another reason I like php apps [05:49:21] more seriously, there are people that have installed it with stock ubuntu ruby [05:49:34] and even stock ubuntu gitolite [05:49:42] you put the code in a directory, you install some debs from apt, and you set up apache [05:50:14] it's not like php is very popular among sysadmins though :) [05:50:24] I'm perfectly happy using php apps [05:50:31] but we've had that discussion before [05:50:38] I wouldn't use it to develop system stuff [05:51:04] well, I guess I should take that back, I did write OpenStackManager, and I guess that's at least a little system related :D [05:51:06] Ryan_Lane: i think this is your problem: http://i.imgur.com/zh4hG.png [05:51:15] ori-l: hahaha [05:51:32] oh, I'm nearly done with the OSM upgrade [05:51:33] you need to run before you jump [05:51:42] Ryan_Lane: yay! [05:51:56] paravoid: and it supports multiple regions now : [05:51:57] :) [05:52:15] one of you has to change names eventually [05:52:19] (OSM or OSM) [05:52:22] ;( [05:52:32] how are regions are zones different? [05:52:32] jeremyb: use your context detector :) [05:52:42] zones are within a region [05:52:42] aude: help? [05:52:51] they use the same services [05:52:51] but you said we won't have zones now? [05:52:59] no. just regions [05:53:21] in the folsom release we can do "cells" within a region for better separation [05:53:29] paravoid: ganeti has neither? just make a separate cluster if you need some isolation? [05:53:32] we can technically use zones, but they are limited usefulness [05:53:43] but i guess no cross cluster migration [05:53:47] regions actually better describe what we're doing [05:53:53] jeremyb: ganeti has nodegroups, but it's a different concept than zones [05:53:56] and offer the proper service separation [05:54:07] paravoid: does grnet use webmgr? [05:54:16] no, we made our own [05:54:18] paravoid: the new scheduler also allows "hints" [05:54:25] webmgr was actually forked from an early version of our webmgr [05:54:36] orly [05:54:37] so, you can pass a hint to the scheduler to tell it that it should run an instance on a particular set of hardware [05:55:45] paravoid: btw, asher is thinking we probably need 3 database servers per datacenter for the replicated databases [05:56:03] I'm thinking maybe we can re-use the old hardware as user databases [05:56:32] I was thinking of using it as a staging cluster for testing upgrades otoh :) [05:56:38] heh [05:56:41] probably a good idea too [05:56:49] we have nothing to test new Openstack or OSM releases on right now [05:56:54] or new features [05:56:57] yeah [05:57:01] I test it in labs [05:57:07] but it isn't 100% testing [05:57:10] right [05:57:28] what do you mean by user databases? [05:57:33] the toolserver stuff? [05:57:34] DaaS [05:57:49] hahaha you're so trendy :-) [05:57:57] meh, it's an easy way to describe it [05:58:43] so... do you envision any point in the future when bare metal can be part of labs? as it's own machine type(s). it's in your project VLANs and reset to clean state when you're done with it [05:58:53] jeremyb: haha [05:58:55] :D [05:59:02] paravoid *just* brought this up [05:59:11] s/VLANs/security groups [05:59:12] :-) [05:59:24] paravoid: btw, essex has support for bare-metal [05:59:27] Ryan_Lane: here? i haven't read *everything* [05:59:30] or atleast the code indicates that [05:59:37] orly [05:59:40] jeremyb: nah, in ops list [05:59:43] ah [05:59:51] yeah, sorry about that [05:59:58] maybe I should had sent it to the labs list instead [06:00:04] heh [06:00:07] always getting confused between the two [06:00:39] Ryan_Lane: any idea why DNS doesn't seem to be created for new instances lately? [06:00:43] is it the 500s? [06:00:45] yes [06:00:48] anyway, so remind me what's the labs version of trainwreck? [06:01:03] and how do we fix it when it happens? ldapmodify? [06:01:04] will all mysqlds have a copy of s4 (commons)? [06:01:04] I need to do a live-hack in OSM for that [06:01:07] paravoid: yeah [06:01:12] oh dear [06:01:29] I should also write a maintenance script to fix that [06:01:30] ugh [06:01:58] we'll be using the OSM dns code after the upgrade too [06:02:15] whose DNS code now? [06:02:21] it's going to take me too long to do the code changes and switch to the openstack dns stuff [06:02:42] I think that's better [06:02:50] one step at a time [06:03:04] also allows us for rollback, if we manage to untangle the ppa bullshit [06:03:36] yes [06:03:58] well, we'll need to rollback OSM completely if we need to rollback nova too [06:04:22] it's just a matter of switching the tag, though [06:05:01] (btw, Google's 2FA works via SMS by default, the OATH stuff is optional) [06:05:02] Ryan_Lane: http://ganglia.wmflabs.org/latest/?r=week&cs=&ce=&m=load_one&s=by+name&c=dumps&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 doesn't look so bad ? [06:05:12] paravoid: eh? I've never seen the SMS stuff [06:05:20] paravoid: ugh, really? [06:06:52] yep [06:06:55] SMS or voice call [06:07:02] voiceprint? ;) [06:07:53] You’ll first need to set up your phone number to receive codes via SMS text message or voice call. If you have a smartphone, you can later download an app that allows you to generate codes without text messages and even without cell service. [06:07:59] https://support.google.com/accounts/bin/answer.py?hl=en&answer=185839&topic=1056283&ctx=topic [06:08:29] I've never even seen this [06:08:43] I set up 2FA when it was first released, as well [06:09:24] it was definitely like that when I set it up for my WMF account recently [06:11:35] paravoid: you mean for labs or something else? [06:11:44] no, google [06:11:48] paravoid: ah, I see [06:11:50] for google apps for wikimedia.org ? [06:11:50] gmail [06:11:52] yes [06:11:53] right [06:12:34] associatedDomain: i-000003a3.pmtpa.wmflabs [06:12:34] associatedDomain: gitlab.pmtpa.wmflabs [06:12:34] dc: i-000003a3 [06:12:40] hmmm, does pdns need anything more than that? [06:12:45] arecord [06:12:51] ah! [06:12:52] that's the only thing that should be missing [06:13:03] but i-000003a3.pmtpa.wmflabs works [06:13:12] gitlab isn't [06:13:21] and indeed there's no arecord [06:14:00] the way the job is written in OSM, it's a problem [06:14:13] if it gets a 500, it thinks the instance is actually gone [06:14:17] I need to check the return code [06:14:24] I'm not totally sure how to do that using the ec2 api [06:14:32] the openstack api is sooooooooo much better [06:14:57] * jeremyb has to assume the OS API was a public collaboration? and ec2 was not? [06:15:18] ec2 is controller by amazon [06:16:11] sure [06:19:39] interesting. it's possible to disable 2FA if you are logged in [06:19:44] that seems dangerous [06:20:07] my implementation is a lot stricter [06:22:56] additional info: The entry dc=i-000003a3,dc=pmtpa,ou=hosts,dc=wikimedia,dc=org cannot be modified due to insufficient access rights [06:23:01] hmmm? [06:23:19] which user are you using? [06:23:27] faidon [06:23:29] heh [06:23:39] you probably don't have access for that :D [06:23:48] let me add you [06:23:53] heh [06:23:55] thanks [06:24:06] isn't it a bit late for you to be working? [06:25:47] ok, you should be good to go now [06:26:10] thanks! [06:27:00] yw [06:27:04] working now? [06:28:11] paravoid: and yeah, a little late :) [06:28:15] isn't it a little early for you? [06:28:21] or did you just not go to sleep? [06:28:23] oh well :) [06:28:26] no I did [06:28:29] for a few hours [06:28:42] not many, I'll probably take an afternoon nap [06:28:46] hahaha [06:28:59] I was seriously messed up, so I'm trying to get my schedule straight :/ [06:29:30] I was sleeping at 7am for two or three days, it's like being jetlagged :) [06:29:36] heh. you always stay up way too late [06:29:58] I've always drifted by an hour or two every day [06:30:04] if I don't put an alarm clock [06:31:25] but now I can work at any point during the day and sometimes it actually helps to coordinate with you guys [06:31:44] yeah [06:31:48] so there was no reason not to drift, until it became too much :) [06:34:34] hm. google 2FA also lets you see your tokens again [06:34:40] the scratch ones [06:34:43] I should probably allow that [06:34:53] in case someone forgets to write them down the first time [06:35:09] it also allows you to regenerate them, which would also be nice to allow [06:35:42] oooo. in essex you can group your floating IPs into pools [06:35:52] hrmmm, so tell me again what do the {sys,net}admin global roles get you? [06:36:04] i think you once told me they weren't needed for what i thought they were [06:36:26] jeremyb: sysadmin lets you create/delete instances, manage sudo policy, manage puppet groups [06:36:46] netadmin lets you manage floating IPs, security groups, and dns [06:37:08] Ryan_Lane: did you see the global part? [06:37:19] global roles give you nothing [06:37:27] why do they exist then? [06:37:35] in the deprecated ldap auth you had to be in the global role to be in the local role [06:37:39] besides cloudadmin. that one i think i know [06:37:42] in essex the global groups are gone [06:37:45] ok [06:37:45] including cloudadmin [06:37:49] oh [06:37:50] I'm killing them all off :) [06:37:59] wiki groups will manage it all [06:38:07] hrmmmm [06:38:12] only local roles will exist [06:38:13] are wikigroups exposed in ldap? [06:38:30] you can sync wiki groups with ldap groups [06:38:33] we aren't right now [06:38:46] anyway, essentially they're about as useful as i thought, just missed the deprecation part [06:38:59] yep [06:39:04] it's also an interface I can kill! [06:39:05] \o/ [06:39:24] I can't tell you how happy it makes me to delete code [06:40:05] Ryan_Lane: so... on instance creation have an option to specific puppetmaster hostname. (which can in turn be your own ::self instance). objections? [06:40:27] (maybe even have a per-project default) [06:40:31] that sounds dangerous [06:40:36] why? [06:40:45] specify the server itself? [06:41:11] well, either way, it won't work [06:41:15] why? [06:41:21] because the instance needs to build itself [06:41:27] so what? [06:41:32] which means it needs to talk to the centralized puppetmaster [06:41:38] how would it build itself from itself? [06:41:41] no... [06:41:53] you'd need to have an existing ::self instance already up [06:42:05] and that instance could be a master for other nodes [06:42:11] ::self means it runs on 127.0.0.1 [06:42:26] ok, well some small variant of ::self then [06:42:37] I'd prefer we just move to modules [06:42:45] then we can use environments for this [06:43:25] ori-l: does not merge cleanly [06:43:27] care to rebase? [06:43:31] (sorry for the lag) [06:43:36] did the rebase button not work? [06:43:49] it doesn't [06:43:50] conflicts. [06:43:52] :( [06:44:03] Ryan_Lane: as long as people can't get code from local working directory into puppetmaster without first going through review or approval i think there will be problems. would you allow that with environments? [06:44:20] jeremyb: remote branches would be environments [06:44:27] (push is fine. rubberstamp by someone else is not really) [06:44:45] it could allow direct push from group members who are in the project [06:44:55] sure... [06:47:39] * jeremyb still wants to be able to specify custom puppetmaster on creation [06:50:01] why? [06:51:02] Ryan_Lane: say i want to test a different puppetmaster version or config? [06:51:43] seems like a simple way to let users administer a piece of infrastructure that doesn't really need to be centralized [06:51:54] most people will still use the centralized one i guess [06:52:11] but it won't be the only optiopn [06:52:13] option* [06:52:14] ori-l: nevermind, fixed it myself. you should be all set now (modulo puppet propagation delays) [06:52:27] it sounds like a recipe for instances that don't build to me ;) [06:52:53] Ryan_Lane: that's another thing... better acess to console when it's building (or not) [06:52:56] Ryan_Lane: so, I have some other work queued up, but nothing labs-related [06:53:07] Ryan_Lane: so, until you're done with the OSM/openstack stuff [06:53:11] * Ryan_Lane nods [06:53:14] no worries. [06:53:21] Ryan_Lane: I was thinking of having a look at the exim labs thing [06:53:24] until we're ready for upgrade... [06:53:29] yeah, exim labs thing would work [06:53:31] *rock [06:53:39] both :) [06:53:43] indeed [06:55:00] but yeah, in a day or two [06:55:11] Ryan_Lane: btw, seen the openstack thread "LBaaS IRC meeting notes" [06:55:13] ? [06:55:17] have to manage the work queue a bit :) [06:55:21] jeremyb: didn't see the meeting notes, no [06:55:31] I'm interested in the service [06:55:41] Ryan_Lane: also, I'd like to have a short talk today (after you sleep :) or tomorrow [06:55:46] about the toolserver stuff [06:55:48] ok [06:56:01] asher will be working on the database replication [06:56:05] it'd be nice to get some context before we get into the meeting :-) [06:56:15] the meeting is thursday [06:56:32] I know, that's why I said a short chat "today or tomorrow" [06:56:34] ah [06:56:35] right [06:56:36] heh [06:57:00] ah, timezone diff [06:57:02] sorry about that [06:58:09] paravoid: isn't it nice to live in the same city that your TZ is named for? ;) [06:58:24] http://i.imgur.com/Wvdv4.gif [06:58:52] lol!/ [06:59:27] Ryan_Lane: errr, that's not related to anything in particular, right? [06:59:37] jeremyb: no, it just made me laugh :) [06:59:37] * jeremyb is also getting tiredish [06:59:42] k ;) [06:59:50] next time someone says something really stupid to me, I'm going to send them that [07:00:27] Ryan_Lane: this mediawiki-l thread is begging for you ;-P [07:00:35] which one/ [07:00:44] errr, wikitech [07:00:48] api one? [07:00:50] but it should be on mediawiki-l [07:00:51] yeah [07:00:53] heh [07:01:00] Ryan_Lane: DANE was RFCed today [07:01:04] DANE? [07:01:29] TLS verification over DNSSEC [07:01:39] o.O [07:01:51] you put TLSA records in your DNSSEC-signed zone and the browser verifies *that8 [07:01:57] instead of using a CA [07:02:04] ahhhh [07:02:33] http://tools.ietf.org/html/rfc6698 [07:03:05] fiiiiinally [07:03:26] paravoid: played much with monkeysphere? [07:03:30] that should be useful [07:03:37] jeremyb: not at all [07:03:49] now we just need to actually use dnssec [07:03:49] Ryan_Lane: if only DNSSEC wasn't as broken as it is... :/ [07:03:49] :) [07:03:53] hahaha [07:03:57] yeah [07:04:11] "let's trust a million clueless registrars to handle crypto, what could possibly go wrong!" [07:04:26] had it's own talk at dc10. upstairs in shapiro. i think i've watched the video (and was there for part of it maybe?) [07:04:47] but of course it's been mentioned other times [07:05:45] I've heard much about it but haven't played with it yet [07:05:46] * jeremyb does like the concept i think. at least the parts that I understand (monkeysphere) [07:20:23] paravoid: hey, thanks for the merge! [07:20:41] sure [07:20:46] yvw :) [07:21:32] do i get to find out the secret handshake now? [07:21:42] :) [07:21:54] ori-l: DH [07:21:55] Change on 12mediawiki a page Talk:Developer access was modified, changed by Kozuch link https://www.mediawiki.org/w/index.php?diff=569747 edit summary: /* There was either an authentication database error or you are not allowed to update your external account. */ ::Yes it works now. Thansk for help.--~~~~ [07:22:20] DH? [07:22:42] Change on 12mediawiki a page Talk:Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=569748 edit summary: /* There was either an authentication database error or you are not allowed to update your external account. */ re [07:23:19] ori-l: Diffie–Hellman [07:23:53] i use rot13 myself [07:24:32] it's a shared key negotiation algorithm not a cipher [07:24:48] jeremyb: i know. it's late and my jokes are lame. [07:24:59] ;) [07:25:09] * jeremyb isn't so great with the jokes himself [07:25:18] may you wait for the F train forever [07:25:22] for poking fun at me [07:25:22] :P [07:25:23] but DH is literally a secret handshake! [07:25:40] bev-y: lol. how did you know where I live? ;) [07:26:25] (there's a bunch of options in this neighborhood) [07:27:23] huh, i wonder what broke aude. we're both in newark... [07:27:57] the other options are more or less reliable [07:28:07] if i curse the F it has a better chance of materializing [07:28:22] uhuh [07:28:24] and the G? [07:28:33] the G is not a subway line [07:28:36] it's a practical joke [07:28:49] i fell for it once, waiting on the platform for a good hour or so before someone told me it was a gag [07:29:40] i moved from one G stop to another. but both places were at stations that also had other lines [07:30:54] where are you at in park slope? [07:33:39] *sigh* [07:33:41] ruby people [07:34:13] hrmmm, should i look at lucus's blog? ;) [07:34:21] I run "gem install bundler; bundler ..." [07:34:27] and now it fetched 5 git repos [07:34:39] haha [07:34:44] and it says "Installing foo (1.0)" [07:35:02] for FUCKING SEVENTY FIVE GEMS [07:35:21] make that 78 [07:35:27] and what the fuck is "rubypython"?!?! [07:35:42] RubyPython is a bridge between the Ruby and Python interpreters. It embeds a running Python interpreter in the Ruby application’s process using FFI and provides a means for wrapping, converting, and calling Python objects and methods. [07:35:47] holy mother [07:36:37] hahahahahahahahahahahahahaha [07:36:43] * Ryan_Lane feels vindicated [07:36:50] http://rubypython.rubyforge.org/ [07:36:54] seriously, look at that [07:36:58] I'm speechless [07:37:04] it compiles into javascript [07:37:35] what are you installing to begin with? [07:37:39] RubyPython.start # start the Python interpreter [07:37:40] cPickle = RubyPython.import("cPickle") [07:37:40] p cPickle.dumps("Testing RubyPython.").rubify [07:37:40] RubyPython.stop # stop the Python interpreter [07:37:43] WHAT THE FUCK [07:37:46] :D [07:37:52] paravoid: still want to use that? [07:38:02] 78 gems? seriously? [07:38:20] including a python bridge? :) [07:38:42] and I was wondering why the instructions said "apt-get install rubygems python-pip" [07:38:46] I was like "pip? really?" [07:38:51] "what for" [07:38:58] so not 78 dependencies [07:39:00] * jeremyb spies http://wiki.debian.org/Teams/Ruby/RubyExtras/OnRubygems [07:39:01] those are just the ruby ones [07:39:03] oh dear fucking lord. we get gems and pip? [07:39:07] Installing libv8 (3.3.10.4) [07:39:07] Installing rubypython (0.6.2) [07:39:13] yay, javascript too! [07:39:15] wow [07:39:29] * ori-l waits patiently for npm [07:39:36] paravoid: you realize you aren't selling me on this, right? :) [07:39:47] pip -> pyjamas -> npm is my bet [07:40:07] you should have said "oh, it's so easy to install, and it wraps it all up in a neat bundle" and just lied. heh [07:42:59] ok, here's the post i was thinking of [07:43:10] can't seem to find the summary followups [07:43:53] gah, no paravoid [07:43:56] heh [07:44:03] Ryan_Lane: what is he installing? [07:44:08] gitlab [07:44:12] ohhh [07:44:18] * jeremyb forget to paste the link [07:44:22] http://www.lucas-nussbaum.net/blog/?p=617 [07:44:36] yeah, I read that ages ago [07:44:54] good ;) [07:45:02] jeremyb: schlaff ;) [07:45:19] aude: who? ;) [07:45:45] i think there's someone on the street that's locked out [07:45:51] i still can't ssh maps-osmmapnik.pmtpa.wmflabs [07:46:06] or attempting to have a conversation with someone inside a building [07:46:07] but can do ssh i-0000039b.pmtpa.wmflabs [07:46:15] jeremyb: really? [07:47:07] aude: so there's a performance issue on labs with the way nodes are pruned (or not pruned) [07:47:16] ok..... [07:47:17] aude: which causes 500's periodically [07:47:26] aude: and in turn causes this DNS problem [07:47:35] * aude nods [07:47:37] aude: needs manual fixing [07:47:38] ls vendor/bundle/ruby/1.9.1/gems |wc -l [07:47:41] 98 [07:47:41] ok [07:47:44] paravoid: want to fix another? [07:47:50] in LDAP [07:47:53] fix what? [07:47:59] oh sure [07:48:01] which one? [07:48:02] i still can't ssh maps-osmmapnik.pmtpa.wmflabs [07:48:02] paravoid: DNS missing for i-0000039b [07:48:06] but can do ssh i-0000039b.pmtpa.wmflabs [07:48:44] up to 98 now i see [07:50:10] fixed [07:50:45] paravoid: amazing, thanks! :) [08:43:47] Ryan_Lane: not sure I followed your question about user name [09:00:16] ah, found what libv8 was for [09:00:21] https://github.com/cowboyd/therubyracer [09:00:30] DESCRIPTION [09:00:31] Embed the V8 Javascript interpreter into Ruby. [09:00:43] ori-l, Ryan_Lane ^^^ [09:06:05] which in turn is used by https://github.com/sstephenson/execjs/ which also supports node.js besides therubyracer [09:06:09] ori-l: prophecy confirmed. [09:11:04] OrenBo: it was meant for ori-l, he typoed [09:11:10] OrenBo: (with a tab key) [09:11:32] ok thanks! [09:11:33] paravoid: so, npm? [09:11:45] OrenBo: sure [09:12:13] jeremyb: how are you doing these days ? [09:13:19] OrenBo: pretty good, you? [09:13:45] OrenBo: you've been "Outbound @JFK" for quite some time i see. i hope that's a mistake! [09:13:52] ;) [09:43:35] skype ? [10:23:00] OrenBo: jabber [15:41:40] Change abandoned: Jens Ohlig; "(no reason)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/10567 [16:39:28] paravoid, do you know if puppet imposes a timeout on system calls? [16:40:09] erm, you mean Exec? [16:40:26] yeah [16:40:54] yeah, there's a timeout [16:41:03] I don't remember how much is it [16:41:25] d'you think I can override it for a specific exec? [16:41:27] * andrewbogott googles [16:43:22] yeah [16:43:25] timeout => NN [16:43:33] but you really shouldn't do long tasks in puppet [16:45:28] This is for a 'git clone' which shouldn't be a long task but sometimes is. [17:27:51] andrewbogott: well, to clone mediawiki it takes quite a while [17:28:05] yep, that's why I'm tuning the timeout. [17:29:01] Hmm [17:43:30] hey Ryan_Lane, can i bug you for an additional external ip for project editor-engagement? it's for a machine used by aaron halfaker (contract) in minnesota, he needs to have researchers be able to browse it via http [17:45:05] done [17:45:21] Ryan_Lane: you're awesome, thanks for being responsive to annoying requests :) [17:45:33] yw [17:45:46] <^demon> Ryan_Lane: So, some guys are looking at writing a plugin for gerrit to integrate the java-based repo browser GitBlit into gerrit. And to actually make it respect gerrit's acls \o/ [17:46:06] respect it to what degree? [17:46:12] including drafts? [17:46:16] <^demon> Presumably, yes. [17:46:56] actually looks pretty nice [17:47:00] ^demon: I hear github has a nice interface [17:47:01] * Damianz runs [17:47:07] <^demon> Damianz: Github can suck it. [17:47:15] Damianz: for our use-case I don't think it does [17:47:32] speaking of gerrit... :-) [17:47:34] It sucks for forcing code review which getting does well in an ugly way [17:47:36] when you want to only search our repos for something, how do you do it? [17:47:39] <3 github for oss stuff [17:47:55] when you want to get a federated view of our repos, how do you do it? [17:48:18] github is fine for individual repos, but not so great for hosting a ton of your own reos [17:48:20] *repos [17:48:48] I agree gerrit is better in many senses, however now that everything is forced to be reviewed self-registration would bring more contributers which is more github like. [17:48:50] hi Ryan_Lane, could I get public IP adress for an instance? I am installing mediawiki on it with written extension. I tried to allocate publice ip through "manage addressess" but it does not work. [17:49:11] Like if I want to fix x on a github repo it takes me <5min, for us it might take 3hours for an account, 2hours figuring out gerrting them 5months waiting for review [17:49:20] MichaelShavlovsk: are you ready to demo to the public, or are you just needing to access it for development reasons? [17:49:23] (though in facebook's case, 5months would be good for a pull request to be merged!!) [17:50:23] <^demon> You know, you can polish a turd, but it's still a turd. [17:50:27] <^demon> s/turd/github/ [17:50:56] s/github/java/ [17:51:05] Though actually... ruby, meh worse than java [17:51:10] ^demon: I think I'd like gitblit more than gitlist [17:51:12] Ryan_Lane: well, now for development reasons, so I need that another memeber of the project could test it online [17:51:20] <^demon> Ryan_Lane: I do too, now that I'm playing with it. [17:51:25] <^demon> And it seems more robust/stable [17:51:32] MichaelShavlovsk: does the other member have labs access? [17:51:36] yes [17:51:42] /usr/bin/rrdtool: symbol lookup error: /usr/lib/libpangocairo-1.0.so.0: undefined symbol: cairo_ft_font_options_substitute [17:51:43] !socks-proxy [17:51:43] ssh @bastion.wmflabs.org -D ; # [17:51:49] * Damianz goes to kick the shi* out of rrdtool [17:51:55] MichaelShavlovsk: you should use a socks proxy, or port redirection [17:52:26] public IPs are limited, so it's best to only use them when you are ready to do a public demo [17:52:39] <^demon> I find it mildly ironic that GitBlit's source is on GitHub. [17:52:44] <^demon> ;-) [17:53:46] GitBlit does look sorta nice [17:54:28] Ryan_Lane: okay, thank you, is there manual how to do port redirecton? [17:54:40] @search redirect [17:54:40] No results were found, remember, the bot is searching through content of keys and their names [17:54:45] @search port [17:54:46] Results (Found 5): ryanland, load, socks-proxy, port-forwarding, report, [17:54:52] !port-forwarding [17:54:53] https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_services_using_port_forwarding [17:55:14] <^demon> !ryanland [17:55:14] in case you want to get to wonderfull land of labs use portal we call bastion, you will see amazing world where vm's runs happily and nfs friends with ntfs, puppets are fresh and gerrit is ugly :O [17:55:22] <^demon> Um, what? [17:55:32] Ryan_Lane: thanks [17:55:33] nfs can diaf [17:55:49] gerrit is sorta pretty now [17:55:52] <^demon> !ryanland del [17:55:52] Successfully removed ryanland [17:55:53] needs a better icon though [17:56:06] <^demon> !ryanland is in case you want to get to wonderfull land of labs use portal we call bastion, you will see amazing world where vm's runs happily and nfs friends with ntfs, puppets are fresh and gerrit is less ugly than it was before :O [17:56:06] Key was added [17:56:19] <^demon> Damianz: We wanted a fire-breathing bear ;-) [17:56:32] <^demon> "Gerrit the Grizly" [17:56:38] Can we not use the oatmeal bear? ;) [17:58:20] we could ask the oatmeal to make one. heh [17:58:57] hi ^demon, could I get git repo for a project, I filed request at http://www.mediawiki.org/wiki/Git/New_repositories/Requests [18:00:08] <^demon> Is there any existing history you're wanting to import, or is this a brand new extension you're starting? [18:01:16] this is a brand new extension [18:02:27] <^demon> Even easier :) Just give me a moment. [18:02:34] thank you [18:03:38] <^demon> You're welcome. Ok it's done and ready for you. The repo is named "mediawiki/extensions/BlameMaps" [18:04:32] cool! [19:10:38] Ryan_Lane: Hope you remember Syslog Collection project :) [19:10:59] of course [19:11:55] Exactly how many log rsyslog clients are we talking about? [19:12:21] I'm currently testing logstash according to your advice. [19:13:20] There are some glitches, but I'm contstantly in touch with logstash folks on irc. [19:16:35] up to 2000 [19:16:42] maybe double that [19:21:19] OMG, we will eventually have to do load balancing etc. Right? [19:21:19] I'm a little concerned about the memory usage, since its a java app. But logstash folks are saying that it scales just fine with around 100 clients. [19:22:39] heh 100? :D [19:23:16] that's a really low number [19:27:03] Yes! Seems like we would need 20 logstash servers for 2000 clients!! [19:32:34] yeah, that wouldn't do [19:32:40] which channel is it? #logstash? [19:34:50] Yes, you are there already :) [20:08:07] Ryan_Lane: What do you think from what they just said at #logstash? [20:09:45] seems it's still worth testing [20:17:02] Yes, after all, it's really easy to setup. [20:17:11] As of now, I'm upto fixing my logstash init script. [20:17:17] Then, I'm planning to do dump hug number of logs/sec to test how it scales. [20:17:24] In the mean time, it would be really great if I get a hand from a PHP expert to implement your private/public saved searches (I'm no good for that :) ). [20:17:38] I'm thinking of spreading things accross: 1 machine for logstash, 1 machine for elasticsearch, 1 machine for apache2 which will provide us front end for saved searches, gather the query and direct to logstash server. What do you think? [20:26:59] andrewbogott: I think logstash is ruby [20:27:49] * andrewbogott lacks context [20:32:29] whoops [20:32:33] sorry, wrong person [20:32:39] adminxor: I think logstash is ruby [20:34:13] * Reedy hands Ryan_Lane a stabby knife [20:34:45] Reedy: this is *jruby* at that :D [20:34:53] the good thing is, it doesn't require a billion gems [20:35:03] oh yeah, you've told me about that before [20:35:09] I'm not sure what I hate more of the two [20:35:13] (yes, I know where you stand) [20:35:24] heh [20:35:25] Hello! I'm your friendly footer. If you're actually reading this, I'm impressed. [20:35:28] hahahahahahaha [20:35:35] ? [20:35:41] from logstash.net [20:36:02] heh [20:36:09] looks nice... [20:36:15] from a scaling POV, I'm not sure how well it'll work [20:36:17] gitlab looked nice too :P [20:36:27] but, the people in the channel had good responses when it came to scaling [20:36:31] so we can at least check it out [20:36:32] <^demon> Ryan_Lane: jruby? [20:36:36] <^demon> Ruby written in java? [20:36:36] Ryan_Lane: did I tell you about gitlab's execjs dependency? [20:36:36] and see how quickly it falls down [20:36:45] ^demon: ruby running in jvm [20:36:46] paravoid: no. heh. what's that? :) [20:36:59] Ryan_Lane: ExecJS lets you run JavaScript code from Ruby. It automatically picks the best runtime available to evaluate your JavaScript program, then returns the result to you as a Ruby object. [20:37:07] oh. my. fucking. god. [20:37:13] burn it with fire [20:37:20] supports therubyracer (libv8), therubyrhino (mozilla rhino) and... [20:37:23] *drumroll* [20:37:26] Node.js! [20:37:28] Why aren't we using that instead of lua? [20:37:30] hahahahahahahahaha [20:37:56] Reedy: using what? [20:38:04] that's in addition to rubypython of course [20:38:09] <^demon> Ryan_Lane: Can I have a labs project to DOS their site? [20:38:17] paravoid: you're killing me [20:38:35] ^demon: gitlabs requires 78 gems, rubypython, and execjs [20:38:40] 98 [20:38:48] *98*! hahaha [20:39:01] incl. a separate mysql library [20:39:07] mysql2 [20:39:08] <3 [20:39:12] ruby bindings that is [20:39:19] <^demon> Are they based here in town? I want to find them and whack them over the head with a bunch of steel pipes. [20:39:58] well, to be fair, most of the dependencies make sense (if only they weren't gems...) [20:40:06] yeah [20:40:11] like net-ldap [20:40:16] ^demon: Be careful with the pipes, lcarr will stab you ;) [20:40:17] yay, works with ldap out of the box [20:40:32] paravoid: so does gerrit and phabricator :) [20:40:44] <^demon> Damianz: Leslie loves me, I won't get stabbed :) [20:40:55] Leslie loves anyone that takes her cookies though :P [20:41:07] Ryan_Lane: can we pay the gitlab person to just do frontend stuff from now on? [20:41:11] like, for us? [20:41:15] for gerrit or whatever? [20:41:18] heh [20:41:26] the meeting with gerrit is going pretty well so far [20:41:26] it's really nice [20:41:26] paravoid: Only if he fixes the damn ssh key ldap thing first :( [20:42:19] Must stop eating dry cereal and get real food [20:42:20] but yeah, not terribly excited for its setting up part [20:42:40] paravoid: that also means maintenance is going to be hell [20:42:47] well yeah, same thing [20:42:56] upgrading gerrit is dropping in a new jar, updating the database and starting gerrit [20:43:04] *war [20:43:10] well, they use bundler and all that [20:43:15] it was trivial to get the gems etc. [20:43:21] in that sense I don't see much difference from gerrit [20:43:35] <^demon> Ryan_Lane: I made gerrit-build on labs and installed maven so we can build it ourselves :) [20:43:38] but ewww nevertheless. [20:43:57] ^demon: cool [20:43:59] ^demon: can we deploy the qt patches? [20:44:11] or is it too much work? [20:44:30] <^demon> Prolly not too much work, but I *really* don't want to end up maintaining it. [20:44:40] fair enough [20:44:58] <^demon> I'm afraid Qt is gonna just abandon it :( [20:46:04] We need to get Google to buy Nokia/Qt [20:49:57] rubypython can run with jruby too [20:49:59] * paravoid giggles [20:50:47] paravoid: :D [20:53:40] <^demon> You know, if gitlab doesn't use prolog, then it's not a winner in my book. [20:53:52] <^demon> You can't write a winning code review tool if you're not compiling prolog into java. [20:54:10] *g* [20:54:12] hah I had forgot about that [20:54:53] seriously, what's up with that? [20:55:01] <^demon> Best. Idea. Ever. [20:55:23] I hope you're kidding :) [20:55:37] <^demon> No, I'm not. It's really not a bad idea if you dive into why they did it. [20:55:54] <^demon> It actually makes a fairly decent workflow engine. [20:56:43] apparently they forked the prolog->java engine too... [20:56:58] ironically, this sounds a lot like ruby developers :P [21:50:41] \o/ [21:50:52] I'm eliminating like every admin call to nova [21:51:22] in fact, we only need a user that can read, and only for the dns stuff. when we drop that, we can totally eliminate the fake user account [22:13:06] http://www.wikipediaredefined.com/ < interesting but I actually think it looks worse than now [22:14:11] it's slow as hell, that's for sure [22:14:26] It did hit top 10 on hn tbf [22:15:05] yeah, this is terrible [22:15:34] Ryan_Lane: Got a second please? [22:15:42] adminxor: sure [22:15:47] He does, everyone has the same number of seconds every second :) [22:16:22] * Damianz stops being a pretentious douche bag and goes back to looking at bikes [22:16:34] I *much* prefer the athena redesign [22:17:15] <^d> Damianz: Not true, my seconds are actually 1.5 standard seconds long. [22:17:28] ^d: You run java though [22:17:36] NTP has a leap second and you're fucked [22:17:42] Ryan_Lane: point your browser to http://i-0000035e/kibana [22:17:52] It looks cool [22:18:08] <^d> Damianz: Touché [22:18:21] adminxor: you should link to the fqdn ;) [22:18:29] which app is this? [22:18:34] kibana? [22:18:43] oops..my bad [22:19:28] Yes, default interface of logstash is not really helpful. Plus it throws lots of errors which I'm yet to fix. [22:19:35] Kibana works smooth. [22:19:40] ah, this is an interface on top of logstash? [22:19:48] or just another log tool? [22:19:55] ah. alternative browser [22:21:16] It works on top of logstash/elasticsearch [22:21:23] * Ryan_Lane nods [22:22:15] But, close to what you wanted. I will have to poke around a little to get exactly what you wanted i.e save searches and stuff [22:22:34] Also, it lets you save the result on a spreadsheet [22:22:46] you can view live stream of logs too [22:23:06] What's not possible in Open Source world! :) [22:34:03] adminxor: heh [22:34:12] this is ruby as well, right? [22:34:22] how many billion gems did it require? [22:35:14] adminxor: hey, i think your user page is in danger of getting out of date fast ;-P [22:35:17] adminxor: have a look at http://jpmens.net/2012/08/06/my-logstash-and-graylog2-notes/ [22:35:36] it compares graylog2 & logstash [22:36:02] both use elasticsearch and it mentions some elasticsearch web interfaces (incl. kibana) too [22:36:40] Ryan_Lane: kibana is php [22:36:45] Kibana is ruby by the way! [22:37:11] oh? [22:37:23] it says php [22:37:40] what does lsof say? ;) [22:37:42] correct. [22:37:50] okay [22:38:08] logstash is (j)ruby, uses elasticsearch (java) which uses lucene (java) [22:38:15] graylog2 is java, also uses elasticsearch [22:38:22] sorry, just now i saw i meant "kibana is not ruby by the way" [22:38:29] :D [22:38:31] graylog2-web is RoR, but you can use Kibana instead I'd think [22:39:38] Ryan_Lane: paravoid: check this out: http://divisionbyzero.net/article/2012/06/17/central-logging-with-open-source-software.html [22:40:05] "So why use logstash instead of Graylog2? It has to do with the indexes. Graylog2 implements a single index 'graylog2' in the ElasticSearch cluster. This makes the search API fairly simple, as I simply specify that index to search from and give my filter criteria. The downside, this index is ENORMOUS, so simple searches, or unbounded searches could dramatically impact the availability of the entire cluster." [22:44:47] yikes [22:45:45] eww [22:45:49] graylog2 also uses mongo (not a big fan) [22:45:52] +1 [22:46:12] logstash sounds fine for our purposes [22:46:16] hm. now, how do we realistically test logstash? [22:46:25] we can syslog some labs instances to it [22:46:55] eh? [22:46:58] not too many, or we'll kill everything [22:47:09] why wouldn't we just set it up and push all of our logs? :-) [22:47:14] hahahaha [22:47:16] * paravoid takes a hammer [22:47:27] you obviously want labs to die :) [22:47:41] didn't say labs [22:47:42] slashhammer might be more useful, paravoid [22:47:49] paravoid: mongo's NYC fyi [22:47:56] I'd like to test it and puppetize it in labs before we set it up in production [22:48:00] I don't think labs would die, but I think we'll have private data in our logs [22:48:07] so labs is not suitable [22:48:09] ... [22:48:18] I don't mean production logging to labs [22:48:24] I mean labs logging to labs [22:48:34] oh [22:48:41] (sorry, kinda late here) [22:48:43] paravoid: give me another day of time before I can make the logstash instance clean and clear for the log bombs [22:48:44] heh [22:48:46] no worries [22:49:02] adminxor: we have to make a strategy before actually doing this [22:49:34] I think we should cherry-pick some instances to syslog to logstash [22:49:37] right [22:49:55] bots, deployment? [22:50:03] that should be plenty [22:50:08] maybe wikidata people want in [22:51:06] maybe also log prod SAL and labsconsole SAL to it too [22:51:23] Well, the rsyslogs clients will still point to rsyslog server. rsyslog server would then work as a shipper to the logstash server. Wouldn't that be a better approach? [22:51:25] I think you're being too careful :) [22:51:33] what adminxor said [22:51:37] let's just redirect everything [22:52:14] it's just system logs, I'd be amazed if we get at 1 message per second [22:52:36] dhclient [22:52:39] it's spammy as fuck [22:52:50] because of the low refresh time [22:53:45] We can actually use that dhclient logs to measure loads on the logstash server [22:54:48] later on we can put a filter in logstash conf to filter out from outputs if needed. [22:56:42] I agree [22:56:51] anyway, really leaving now [22:56:53] love your work adminxor [22:56:55] Also, we can write logs locally as well as redirect to the logstash server, just to make sure important logs of the lab machines are not missing out. [22:57:46] paravoid: I did not do anything as such. If it works, credit should go to the folks who created all these incredible tools. [22:58:45] still, thank you [22:58:50] :) [23:08:47] sorry, have to leave now. shall come back at 1.30PM EST tomorrow [23:11:50] errrr, EST -> EDT ;) [23:29:01] csteipp1: Ryan_Lane: do we know any simplesamlphp users? [23:29:18] security release: http://packages.debian.org/changelogs/pool/main/s/simplesamlphp/simplesamlphp_1.9.1-1/changelog [23:29:33] jeremyb: nice :) [23:30:00] jeremyb: faidon does