[00:00:31] so jenkins would run it periodicaly or when some thing changes in GIT [00:00:50] that makes more sense to me now [00:00:57] I'd make it trigger based on a comment from a core committer [00:01:11] we don't want every single commit launching a ton of instances ;) [00:01:33] Ryan_Lane: was morebots ever puppetized? i dare not ask. [00:01:39] hyperon: ;) [00:01:44] although i just did... [00:01:46] It runs on wikitech [00:01:46] still waiting for you :D [00:01:50] well jekins only runs two tasks at the same time [00:01:52] RoanKattouw: no. the one for here [00:01:54] It cannot be puppetized [00:01:56] Oh that one [00:02:01] Yeah that one is fine [00:02:07] it isn't puppetized [00:02:12] it isn't packaged [00:02:12] The original morebots can't be puppetized in the traditional sense [00:02:13] would it be able to get info from the run adn display it ? [00:02:30] OrenDsk: it would run smoke tests against the instances, and report back to gerrit [00:02:33] Ryan_Lane: we packaged it, remember? [00:02:38] oh. right [00:02:47] then it just needs to be puppetized. heh [00:02:56] which i did [00:03:01] oh? [00:03:03] which change? [00:03:04] and then i never got gerrit to work [00:03:09] ah [00:03:37] so once i figure out that, it will be happily puppetized [00:03:44] * Ryan_Lane nods [00:03:52] what issues are you having? [00:03:55] actually, i'll do that after dinner [00:04:03] Ryan_Lane: keys [00:04:05] so you would ping thing and run wgets ? [00:04:16] hyperon: you need to add your key to gerrit [00:04:17] or use phpUnit [00:04:20] it stores the keys separately [00:04:42] Ryan_Lane: which i thought i did, but i'll double check. after dinner. [00:04:46] * hyperon to dinner [00:04:55] OrenDsk: I don't know. we haven't started to implement anything yet [00:06:21] Ryan_Lane: I'd working on making mwdumper playback edits into a wiki for testing [00:06:49] that would be interesting for some tests, yeah [00:07:17] Ryan_Lane: I would be very intersted in seeing how you generate a new instance from jenkins - since it would be simpler than resetting the database etc [00:07:46] you can use euca2ools, or the nova command [00:08:09] or you can call the API directly [00:08:26] of course, jenkins would have to have credentials to do so [00:08:31] you could - I don;t have a clue [00:08:37] and there is that [00:08:51] it's seriously just a command [00:08:51] problem with jenkins is that it is abit open [00:09:28] ok - if you put a sample or two in the labs wiki that would be great [00:09:45] well, the instance somehow needs to build itself too [00:09:54] so, you'd have to have a way of automating that too [00:10:02] yep [00:10:12] which is why I planned on using puppet [00:10:21] but you have figured that out [00:10:39] I'm not ready to start work on this stuff right now [00:10:55] I am doing sevral things as well [00:11:00] also, I'm not ready to let people create instances outside of using the labs console [00:11:31] puppet currently wouldn't work without using labsconsole [00:11:42] we have openstack related work that needs to get done first [00:11:55] Ryan_Lane: I came across something interestig today [00:11:59] oh? [00:12:09] it's called apache Zookeeper [00:12:25] Zoekeeper is very interesting [00:12:39] it allows to control large clusters [00:13:23] I'm not so great at explaining it - but here is the link http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CEYQFjAB&url=http%3A%2F%2Fzookeeper.apache.org%2F&ei=GEo8T8fNIuLm4QTMy9WmBg&usg=AFQjCNHl6-ABSg9AHvtfhp8owX7oIaxKAg&sig2=ask4T9MOE9CgPuZwnIesWQ [00:13:36] dammn google [00:13:48] http://zookeeper.apache.org/ [00:14:17] I don't get it [00:14:36] https://cwiki.apache.org/confluence/display/ZOOKEEPER/ProjectDescription [00:14:58] I know a use case for solr [00:15:16] it allows you to setup a distributed search [00:15:25] with replication [00:15:51] is this middleware for building applications? [00:15:56] is any server goes down you still get to use a replication [00:15:59] middleware [00:16:38] it abstarcs primitive operations for controlling the cluseter [00:16:39] why not just use LVS? [00:17:15] I'm probably just confused about what this actually does [00:17:18] and it lets you manage configuration files [00:17:34] so, this is a system like puppet or chef? [00:17:39] like I said I can't realy explain it too well [00:17:47] or a deployment system like cask? [00:18:01] hell, the documentation does a really poor job of describing it :D [00:18:26] well it has 3 key fetures [00:18:30] it's robust [00:18:43] the documentation is very heavy on the "here's how we do things" but very light on the "this is what it actually does" [00:18:49] its fast for mostly read only taks [00:18:53] I agree [00:19:56] this is how people use solr coud - so I'm looking into it [00:20:31] the current search engine manages config and updates in a much more fragile way [00:20:44] well, we're going to be changing that to puppet [00:21:06] also in prodiction ? [00:21:16] of course. [00:21:26] puppet manages basically everything in production [00:21:36] we use puppet in labs because we use it in production [00:22:07] * OrenDsk bangs head [00:22:16] A zookeeper cluster consists of at least three servers running zookeeper (zk). A client can connect to each zk server and issue read and write requests. Zookeeper guaranties that a write either fails or is consistent and that a read will get you the most recent state. [00:22:17] Zookeeper exposes a filesystem like hierarchy of so called znodes. Every znode can have children but also stores data. The data stored in a znode is assumed to be small (less then 1MB). Clients can subscribe to different events on a znode and will be notified of changes in the znode itself of it's children. [00:22:18] what you said about virtualization ? [00:23:01] Our POC ZooKeeper cluster stores meta data for stuf in hadoop rather than using mysql because mysql cluster sucks doneky balls. [00:23:25] +1 for the use of the term donkey balls [00:24:06] Damianz: pardon my ignorance - but system are you talking about ? [00:24:46] OrenDsk: Un-related to labs. [00:24:55] I understand [00:25:15] I'm just curious what you do there? [00:26:56] Host $stuff, with hadoop nothing right now but hopefully we will be switching over to scribe and using hadoop to pipe logs into for some centralised processing. [00:27:08] preilly: I still don't get what it does, based on that description [00:27:16] is it just a clustering technology? [00:27:17] what about book keeper [00:28:31] they have an channel here [00:29:38] hmm here is a more usefull page http://zookeeper.apache.org/doc/r3.4.3/ [00:29:52] Ryan_Lane: Typical use cases Naming service Configuration management Synchronization Leader election Message Queue Notification [00:30:21] maybe I'm just stupid today [00:30:23] Ryan_Lane: take a look at http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy [00:31:03] * jeremyb waves from a puppet user group mtg ;) [00:31:20] I learn by association generally, and not having an example that tells me "it looks like this" fucks with me [00:32:11] Ryan_Lane: SHINY but will screw your mind trying to figure out how it works? [00:32:58] well, I learn things by comparing them to other things. which works really well for most things (since everything evolves from something else), but every once in a while I run into something I have nothing to compare to [00:33:25] nginx is a web server. it's like apache, which I've already learned, so I just need to learn the differences, right? :) [00:33:48] openstack is like vmware cluster [00:33:48] yeap [00:33:53] etc, etc. [00:34:05] they say is is similar to chubby but here is another link [00:34:06] http://research.yahoo.com/pub/3280 [00:34:21] <^demon> Ryan_Lane: Things do not evolve. They're intelligently designed ;-) [00:34:36] ha ha ha [00:34:43] +2 for ^demon [00:34:46] bahahaha [00:34:51] I don't believe in inteligence - only a blind watchmaker [00:35:05] unless you consider human intelligence and its creations to simply be a process of evolution [00:35:26] wow [00:35:55] meme's are a great example of evolution :) [00:36:02] is this the existencial mediawiki channel [00:36:30] Existential [00:36:35] Ryan_Lane: I read a paper on evolution of languges recently [00:36:43] * Damianz checks if Ryan_Lane is really an ANN based bot that hasn't learned java yet [00:36:52] hehehe [00:37:13] Damianz: I know Java too ;) [00:37:25] the writer tried to claim that languages only devolve [00:37:34] it was quite amusing [00:37:37] I feel sorry for you then :P [00:38:02] Ryan_Lane: you rock! [00:38:20] btw zookeepr is a java app [00:38:27] well, programming languages, as a whole are like each other. you just need to learn the differences [00:38:28] * Damianz thinks he might go to bed as it's 20 to 1 and he's been up for 35 hours [00:38:54] Damianz: u in uk ? [00:39:06] Yep [00:39:46] Zookeeper is driving me nuts [00:40:06] * Damianz looks at OrenDsk in the Zoo [00:41:05] we generally try to avoid java [00:41:19] I guess for my needs it provides config files and does leader managment [00:41:27] python and php tend to win out with us [00:41:35] java is a distant third [00:41:47] though it may be fighting with perl for #3 :D [00:41:57] Ryan_Lane: can the stat servers consume JMX [00:42:01] perl > java :D [00:42:14] I prefer Java to perl. I prefer anything to perl [00:42:21] OrenDsk: stat servers? [00:42:28] ganglia [00:42:30] which stat servers? we have stat servers? [00:42:30] * Damianz lets Ryan_Lane write the openstack stuff in brainfuck [00:42:31] oh [00:42:31] heh [00:42:46] Damianz: ok, ok. I take that statement back ;) [00:43:12] I hust found a way to convert jmx data to a ganglia stram [00:43:13] OrenDsk: I don't know [00:43:21] I hust found a way to convert jmx data to a ganglia stream [00:43:29] why would that be needed? [00:43:36] to send better stats to ganglia? [00:43:54] to send java server stats out [00:43:58] ah [00:44:03] that could be useful [00:44:07] cpu load memory [00:44:36] may even allow to launch indexing [00:44:54] in the near future [00:46:08] anyhow in the age of Jython python runs in java too [00:46:38] Jython makes me like the idea of swing [00:47:19] have you used aspectJ [00:47:54] or Scala [00:48:20] there are losts of Java vairients these days [00:48:51] any how I just finished upating the CV so I better send it and get to bed. [00:49:32] kaldari: so, I think hexmode and/or Reedy, and petan are managing deployment-prep [00:49:50] kaldari: yes, I could help [00:49:55] !project deployment-prep [00:49:55] https://labsconsole.wikimedia.org/wiki/Nova_Resource:deployment-prep [00:49:56] OrenDsk: still here? [00:50:07] yes [00:50:09] kaldari: one of them can add you [00:50:20] kaldari: I think it's mostly being used for 1.19 testing right now [00:50:29] I'm not sure if that's also what you are doing [00:50:45] OrenDsk: I wanted to ask if you could help patrol hewikis ... I think you could read Hebrew, right? [00:50:54] sure [00:51:27] oh great. I think I just broke prototype [00:51:31] oops [00:51:40] wait. no. I didn't [00:51:44] but mysql won't die [00:51:48] so it won't upgrade [00:51:49] * Ryan_Lane sighs [00:52:24] ah. there we go [00:52:32] well, actually, I'd like to get ssh access to prototype first since Andrew's doing some ACW testing there. [00:52:38] ok [00:52:42] lemme get that for you [00:52:53] we should try to move everything to labs as soon as possible, though [00:52:56] hexmode: is there a task fore I can join or something like that ? [00:53:00] I'd really love to kill prototype [00:53:10] and reclaim the hardware [00:53:36] OrenDsk: right now, I'm just getting a rough and ready crew [00:53:51] you speak hebrew? [00:53:56] read that is / [00:54:04] ok. NOW I broke prototype [00:54:25] I hope you're satisfied [00:54:53] prototype killer! :) [00:55:08] technically, whoever set up mysql broke it [00:55:11] OrenDsk: no, I don't. Which is why I need help :) [00:55:13] because they did it wrong [00:55:29] OrenDsk: I asked Amir as well [00:55:35] let me get you his nick [00:57:25] OrenDsk: aharoni is based in Israel, so might be closer to your time [00:58:01] aharoni from the localization team? [00:58:06] yes [00:58:17] he's good [00:59:10] maybe now I should /part #wikimedia-strategy :( [00:59:11] I asked him to help.... hewikisource is going 1.19 tonight, so start checking for reports tomorrow [00:59:11] :p [00:59:18] I really hope prototype comes back up [00:59:24] when was the last time anyone rebooted it? [01:00:13] no idea [01:00:19] oh great. my windows vm won't launch now [01:00:26] that's going to be a problem [01:00:27] it's one of those days [01:00:29] Ryan_Lane: I'd like to cooperate with you in this jenkins testing once it gets going [01:00:37] OrenDsk: that's fine. it'll be a really long time [01:00:47] So how do I get an instance set up on labs? [01:01:11] Maybe this will help: https://labsconsole.wikimedia.org/wiki/Access [01:01:16] we'll looks like writing the solr will take some time too [01:01:17] !account-questions [01:01:17] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [01:01:34] I already have an account [01:02:09] hexmode: do I need special access or I just click on ok boxes ? [01:02:40] oh [01:02:50] I can either make you a project, or you can join another [01:02:54] sorry for killing prototype :( [01:02:59] I'm trying my best to fix it now [01:03:08] oh. it's back up! [01:03:11] \o/ [01:03:47] OrenDsk: Access for this monitoring? Just let me know of any problems you see on Village Pumps. [01:03:59] kaldari: first let me set you up with an account on prototype [01:04:02] ok [01:04:06] since I didn't kill it [01:04:06] cool [01:04:16] !initial-login | werdna [01:04:17] werdna: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [01:04:46] he wiktionary wikipedia wikisource (boring) [01:04:52] kaldari: you already have an account [01:05:01] hmm [01:05:17] let me try again... [01:05:24] want me to reset your password? [01:05:37] he.wikisource is 95% commentries by rabbais [01:05:46] Ryan_Lane: yes [01:05:51] ok. gimme a sec [01:05:59] OrenDsk: tomorrow is only hewikisource ... but if they report something I want to know sooner rather than later. I imagine you could just get by with scanning village pump [01:06:15] ok I'll watch it [01:06:48] tyvm :) [01:07:10] shit. if windows isn't working, I have no clue how I'm going to deal with vmware on tesla [01:07:34] so ready to be rid of tesla [01:07:38] contact Erel Segal if you can [01:09:02] * Ryan_Lane pokes wm-bot [01:09:11] wm-bot: shouldn't you be doing something right now? [01:09:22] Ryan_Lane: And can you create projects? [01:09:27] I can [01:09:32] what project would you like? [01:09:38] want to create an 'ee' project for editor engagement? [01:09:50] how about editor-engagement? :) [01:09:59] that also works, we've been using ee in most places [01:10:05] hah. that user rename I did earlier broke home directory creation [01:10:08] yay! [01:11:12] 02/16/2012 - 01:11:12 - Updating keys for dachim [01:11:13] 02/16/2012 - 01:11:12 - Creating a home directory for kaldari at /export/home/bastion/kaldari [01:11:13] 02/16/2012 - 01:11:12 - Creating a home directory for dsc at /export/home/bastion/dsc [01:11:13] 02/16/2012 - 01:11:12 - Creating a home directory for werdna at /export/home/bastion/werdna [01:11:30] yeehaw! [01:11:41] that's better [01:11:48] I need to fix that bug, it seems :( [01:12:16] \o/ [01:14:46] * Ryan_Lane really doesn't want to fix the ancient home directory management scripts [01:15:30] though editor-engagement means I have to type that out in instance names [01:15:32] oh noes :p [01:19:53] Ryan_Lane: hey this is pretty neat [01:21:30] heh [01:22:20] werdna: no. you instance names can be anything you want [01:22:26] just the project is editor-engagement [01:22:28] ah okay sweet [01:22:41] though instance names must be unique [01:22:46] so I recommend ee- [01:22:48] nod [01:23:03] !log editor-engagement Added kaldari [01:23:12] how fun [01:23:23] 02/16/2012 - 01:23:23 - Creating a project directory for editor-engagement [01:23:23] 02/16/2012 - 01:23:23 - Creating a home directory for kaldari at /export/home/editor-engagement/kaldari [01:23:23] 02/16/2012 - 01:23:23 - Creating a home directory for werdna at /export/home/editor-engagement/werdna [01:23:38] Ryan_Lane: any recommended instance size for just running MW? [01:23:45] small [01:23:49] m1.small [01:23:57] tiny is often worthless [01:24:11] xsmall is also usually ok, but sometimes on the small side [01:24:22] 02/16/2012 - 01:24:22 - Updating keys for werdna [01:24:22] 02/16/2012 - 01:24:22 - Updating keys for kaldari [01:24:28] werdna: you can add/remove anyone you want to that project [01:24:33] @search project [01:24:33] Results (found 3): logging, sudo, projects, [01:24:35] hm [01:24:42] where's the docs for that? [01:24:52] I see the instance in my list now [01:24:52] sweet [01:24:59] yeah, I just added it [01:25:02] no docs, it seems [01:25:12] ah well [01:25:14] so, you can manage project membership via "Manage projects" [01:25:25] yep, I found that and added kaldari already :) [01:25:38] the menu on the LHS is reasonably self explanatory [01:25:49] and is there some way to get a MW install? puppet? [01:25:51] sysadmin lets people create/delete instances, netadmin lets you modify security groups and manage public ips and dns [01:25:52] although it doesn't have a Nova resource (whatever that is) [01:25:56] unfortunately not right now [01:26:02] it's something on the roadmap [01:26:02] Failed to allocate new public IP address. [01:26:15] Ryan_Lane: okay, but I can get apache and all that jazz set up using puppet? [01:26:18] you can have a public IP when you are ready to demo something ;) [01:26:24] the default quota is 0 [01:26:31] werdna: should be able to, yes [01:26:39] can we have a private IP address? [01:26:41] :) [01:26:41] documentation for a lot of this is still severly lacking [01:26:53] kaldari: you get one automatically when the instance is created ;) [01:26:53] DNS too [01:26:55] Ryan_Lane: Ah, I was after some way of being able to test our shit :) [01:27:00] or do I need to use ssh tunnels and so on [01:27:01] @proxy [01:27:05] bah [01:27:06] @socks [01:27:11] @-proxy [01:27:12] ah, I see it now, had to reload [01:27:16] !socks-proxy [01:27:16] ssh @bastion.wmflabs.org -D ; # [01:27:20] really? :( [01:27:22] so, yeah [01:27:25] I recommend a socks proxy [01:27:36] hmm [01:27:36] kaldari: it takes a bit for the instance to fully build. see the console log [01:27:48] when puppet finishes running, it's ready [01:27:50] it may email you [01:27:54] oh [01:28:05] to access the instance, you forward your keys [01:28:07] !access [01:28:07] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [01:28:09] We'll need to use prototype then, we're hoping we can show Howie/Erik/etc our extension in testing without having them set up proxies and all that [01:28:14] ooh it has Nova power now! [01:28:18] oh [01:28:20] <^demon> Ryan_Lane: Thing we could do an apt-get upgrade on formey? [01:28:22] werdna: I can give you a public Ip for that [01:28:29] <^demon> s/Thing/Think/ [01:28:30] sweet! thanks. [01:28:31] ^demon: sure [01:28:54] <^demon> 52 packages can be updated, 39 are security updates :) [01:28:59] ewww [01:29:12] werdna: ok, you can allocate an ip now [01:29:23] when you are ready to demo something, you'll then associate it with the instance [01:29:24] thx :) [01:29:30] and then add a hostname to the IP [01:29:34] use wmflabs domain [01:29:49] and try to use a somewhat descriptive hostname ;) [01:29:50] okay, so now I need to associate that IP with DNS I guess [01:29:59] prototype.ee.wmflabs.org was my plan [01:30:05] hm [01:30:12] subdomains are hard [01:30:26] ee-prototype then :) [01:30:30] heh [01:30:35] DNS system can use some work ;) [01:30:40] I guess we can just do ee.wmflabs.com [01:31:33] <^demon> ee? [01:31:52] <^demon> and try to use a somewhat descriptive hostname ;) [01:31:56] <^demon> ^ That :p [01:32:08] oh. hmm. you *may* be able to do subdomains... [01:32:16] I haven't tried any of this for a while. heh [01:32:23] Ryan_Lane: more importantly, how can I get myself in sudoers? [01:32:25] ee isn't clear enough ;) [01:32:34] oh never mind [01:32:36] I'm on bastion [01:32:48] werdna: you have sudo automatically on your project [01:33:05] <^demon> kaldari: Well I had to ask and I've been around for the past 6-7 years. I think that means its not descriptive enough ;-) [01:33:06] it uses your wiki password [01:33:35] <^demon> First thing 'ee' reminds me of is that old ee.pl extension :p [01:33:40] yeah, I just realised I was trying to sudo on bastion [01:33:47] "This incident will be reported" [01:33:51] could we do editor-engagement.wmflabs.org and have an alias for ee.wmflabs.org? Or is that a pain? [01:34:00] http://xkcd.com/838/ [01:34:34] it's still setting up I think [01:34:36] <^demon> Also, http://xkcd.com/910/ [01:35:02] :) [01:35:12] do you guys just know these xkcd numbers by heart? [01:36:25] PROBLEM Current Load is now: CRITICAL on ee-prototype ee-prototype output: Connection refused by host [01:36:36] Ryan_Lane: also, who runs "betalabs"? [01:36:43] kaldari: no, xkcd is indexed [01:36:53] he puts transcripts in microdata on the page [01:36:57] so in most cases you can just google it [01:37:05] PROBLEM Current Users is now: CRITICAL on ee-prototype ee-prototype output: Connection refused by host [01:37:11] !project deployment-prep [01:37:11] https://labsconsole.wikimedia.org/wiki/Nova_Resource:deployment-prep [01:37:15] werdna: see member list [01:37:45] PROBLEM Disk Space is now: CRITICAL on ee-prototype ee-prototype output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:37:47]