[00:28:52] New patchset: Sara; "First iteration of adding ganglia for labs." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2157 [01:06:49] !accountreq [01:06:49] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [01:07:46] howdy [01:07:58] hello [01:08:01] * Damianz passes Ryan_Lane a cookie [01:08:11] fastily: you wanted an account to run bots, right? [01:08:21] yes sir :) [01:08:58] Hmm 1am, should I go to bed so I'm awake to sort out my spare servers tomorrow or stay up and code all night =/ The possibilities are endless [01:09:22] Ryan_Lane: Could I have an account please? [01:10:18] !account-questions | fastily [01:10:18] fastily: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [01:11:50] shall I respond here? [01:12:03] sure. unless you don't want the world knowing your email address [01:12:08] Only if you're not worried about the cia watching [01:12:09] other members of labs can access your email address [01:12:18] it's fine [01:12:20] okay [01:12:23] 1. Fastily [01:13:11] 2. I don't have SVN, but could my shell account name be under "Fastily"? [01:13:14] 3. fastily@yahoo.com [01:14:39] sure [01:14:55] Ryan_Lane: Would you say the test or production branch of ops/puppet is best for getting an idea of how puppet can be used? Labs seems a little weird to me as it's half puppetized (if that is even a word). [01:16:26] !initial-login | fastily [01:16:26] fastily: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [01:16:30] Damianz: either is fine [01:16:39] Damianz: we occasionally merge production into labs [01:16:52] and also occasionally merge labs into production [01:18:18] Oh cool :) Production seems to change a /lot/ though I guess it's production and will be more labs when we have the production side running. [01:19:04] yeah. the hope is for most changes to happen in labs first, then in production [01:19:17] so changes will go into the labs branch, and get cherry-picked across to production [01:19:27] that, of course, is a long term goal [01:19:57] it's already the case for some services, just not all [01:23:05] Ryan_Lane: Thanks! [01:23:08] yw [01:23:14] oh. let me add you to the bastion project [01:23:50] someone in the bots project should add you there [01:23:59] I would if I could :P [01:24:00] as they should give you info about how stuff works there [01:24:10] are you not in the project? [01:24:20] if you are a member of the project, you can add others [01:24:24] Really? :o [01:24:35] though you should make sure you explain how things workj [01:24:56] It seems i'm in the sysadmin role now too o.0 [01:25:29] Shiny, I like your puppet thing for instances. [01:26:13] Ryan_Lane: If it's not too much trouble, could you help me get set up? I'm a noob programmer and rarely use ssh... [01:26:33] Ryan_Lane: What's the difference betwean a netadmin and sysadmin in a project? [01:27:36] fastily: That wiki article explains how to login to bastion, from there you can ssh into bot instances.... I think bots are going on bots-3 atm but petan is the best person to poke with a semi sharp stick, or supply cookies too. [01:31:39] !access [01:31:39] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [01:31:41] hm [01:31:45] !right [01:31:47] !rights [01:31:47] https://labsconsole.wikimedia.org/wiki/Access#Rights [01:32:36] :) [01:35:11] 02/01/2012 - 01:35:11 - Creating a home directory for fastily at /export/home/bastion/fastily [01:36:12] 02/01/2012 - 01:36:12 - Updating keys for fastily [01:38:58] Ryan_Lane: I think I got it working now [01:39:06] great [01:39:36] thanks! [01:39:48] oh wait... [01:39:59] one more quick question - how to I set up bot instances? [01:42:20] and...is there a java virtual machine on bastion? [01:47:24] hi ryan, [01:47:32] question about branches and gerrit [01:48:00] if you have a local branch foo and you push it to gerrit, how can you prevent that it merges into master without explicit approval? [01:58:43] oh wait, nevermind... I think I figured it out. Should have read the Bots project server logs -.- Thanks for the help [02:01:37] diederik: hm. we block that in the puppet branch somehow [02:02:17] is that something that you can enable? [02:03:47] I'd have to figure it out [02:26:02] because gerrit basically kills the idea of branching :( [02:32:14] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 5.81, 14.32, 7.32 [02:42:14] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.26, 2.22, 3.99 [09:22:32] New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2157 [14:13:09] 02/01/2012 - 14:13:09 - Updating keys for hashar [14:13:12] 02/01/2012 - 14:13:12 - Updating keys for hashar [15:06:05] !log incubator Created new instance incubator-bots3 for offloading bots1 with m1.large configuration [15:06:29] !log [15:06:37] zzz no log bot? [15:08:26] !bug 1 [15:08:26] https://bugzilla.wikimedia.org/show_bug.cgi?id=1 [15:08:31] !log incubator Created new instance incubator-bots3 for offloading bots1 with m1.large configuration [15:10:27] Apparently logging went walkies [15:24:29] So, wikimedia labs, what can it do? Can I, for example, fork enwikisource, install my extension on the fork, and then invite all the enwikisource users over to try it out? [15:26:31] <^demon> Creating instances to test mediawiki extensions sounds do-able. Not sure what the status on the db replication is yet. [15:27:20] OK, so something like this has not been tried before? [15:27:47] <^demon> Well, most of the labs projects are for stuff like analytics and ops stuff so far. [15:27:49] Extension beta testing before production deployment? [15:28:01] <^demon> Well, we used to test stuff on test.wikipedia.org [15:29:03] <^demon> There's a group of people who've been building out wikis that mirror the way the cluster is setup, so they might have an enwikisource already setup that you could test on. [15:29:23] <^demon> I think petan knows more about that. [15:29:44] OK [15:30:25] hello [15:30:43] GrafZahl_: this what you talk about is a proposed goal of development part [15:31:03] Hi petan [15:31:13] operation engineers of wmf are working on that I have no idea what is status now [15:31:31] as for now it is not possible to replicate any production wiki in any simple way [15:31:50] however it's true that we have replicated almost whole cluster on beta.wmflabs.org [15:32:36] unfortunatelly things are becoming more and more complicated, giving access to other people is not so easy as it used to be [15:33:12] is your extensions sheduled to be deployed before major release? [15:33:32] <^demon> I think we'll get the access issue sorted out in the long run. [15:33:35] Nope [15:33:41] <^demon> This is a new project, so there's bound to be some hiccups along the way [15:33:46] In fact, I'm still adding features [15:34:06] ok, in that case the replicated site we made is not suitable for you [15:34:24] we can only install sw which is supposed to be deployed soon [15:34:36] OK, thanks for the info [15:34:44] Does that go for test.wikipedia as well? [15:34:53] that is even worse [15:35:11] installing sw to test.wikipedia require a full code review [15:35:13] <^demon> test.wikipedia is part of the cluster, and is supposed to be the "last staging ground" before things go live. [15:35:24] I see [15:35:49] you would have it deployed to beta.wmflabs definitely sooner than on test.wikipedia but still it should be in final stable version [15:36:08] unstable software is not being tested there [15:36:17] it's pre deployment site [15:36:33] OK, thank you, and I'll keep an eye on the status of labs [15:36:41] Cannothazstaging [15:36:42] !labs [15:36:42] https://labsconsole.wikimedia.org/wiki/$1 [15:36:45] eh [15:36:51] @search conso [15:36:51] Results (found 27): git, labs, instancelist, instance-json, amend, sal, security, sudo, access, stucked, group, pathconflict, terminology, nova-resource, initial-login, manage-projects, rights, projects, docs, instance, address, ssh, documentation, start, link, socks-proxy, requests, [15:36:53] @search console [15:36:53] Results (found 27): git, labs, instancelist, instance-json, amend, sal, security, sudo, access, stucked, group, pathconflict, terminology, nova-resource, initial-login, manage-projects, rights, projects, docs, instance, address, ssh, documentation, start, link, socks-proxy, requests, [15:36:56] omg [15:37:00] @search console.wik [15:37:00] Results (found 25): git, labs, instancelist, instance-json, amend, sal, security, access, group, pathconflict, terminology, nova-resource, initial-login, manage-projects, rights, projects, docs, instance, address, ssh, documentation, start, link, socks-proxy, requests, [15:37:09] Does the beta cluster actually have more than one webserver now? lol [15:37:26] @search mediawiki [15:37:26] Results (found 12): morebots, labs-home-wm, labs-nagios-wm, labs-morebots, gerrit-wm, extension, revision, info, bots, labs-project, openstack-manager, wl, [15:37:31] !info [15:37:31] https://www.mediawiki.org/wiki/WMF_Projects/Wikimedia_Labs [15:37:38] here we go [15:37:42] there is a status tracked [15:37:57] Dantman: yes [15:38:01] oh sorry [15:38:04] Damianz: yes [15:38:22] why :P [15:38:36] we have -squid -web -web2 [15:38:40] for web [15:40:02] :) [15:40:39] Damianz: is it slow? [15:40:47] because it loads faster than production here [15:41:04] if it was I will create more [15:41:26] I was thinking more from the other day when it did in a pile of oom. [15:41:34] ah [15:41:37] yes [15:41:38] Though it does seem fater, tbf though it's not getting db raped. [15:41:52] no sql is mostly idle [15:42:39] I have problems with loading wikipedia but labs are finew [15:42:41] fine [15:42:46] :) [15:55:46] Damianz: you know how to configure nat so I redirect traffic to another instance [16:07:47] hi roan: quick question, what actions on the api can only be done using the POST method? [16:08:25] diederik: The autogenerated help will tell you [16:08:27] Look for what's labelled as "This module only accepts POST requests" [16:08:28] http://en.wikipedia.org/w/api.php [16:08:37] 32 actions on enwiki need post [16:08:42] You can just always post [16:10:03] Reedy: He's asking because he's been analyzing Squid log data [16:10:14] He just PMed me a list of all action= params he found and their frequench [16:10:19] *frequency [16:10:29] petan|wk: Why do you want to NAT it? [16:11:19] It's the same lan so I'd just stick lvs on, far simpler than screwing around with the firewall. [16:16:38] Why is vhn14 playing up now grrr [16:19:12] Errr that was totally the wrong window. [16:34:44] ping petan [16:34:56] petan|wk: [16:39:08] I think he dozed off again [16:39:52] it's oka [16:39:53] okay [16:40:05] i found someone to help me [16:40:17] Either that or he tried to setup lddirector and now has no network access. [17:54:52] aude: hi [17:57:09] hi ryan! [17:57:17] howdy [17:57:21] got some exciting news for you! [17:57:27] .... [17:57:32] (drum rolls) [17:57:36] x2 [17:57:41] .... [17:57:42] oh? [17:58:18] i made a debian package of the new udp-filtering system and the code has been approved by tim and it's well documented and it's all you want from the new filtering system [17:58:54] ah. sweet [17:59:07] how do i pass that deb package to you [17:59:07] ? [17:59:35] is it in svn? [17:59:35] or for that matter, who can help me deploy this on emery? [17:59:43] no, not yet [17:59:46] anyone in ops can help you deploy it [17:59:46] i can put it in git [17:59:58] it should be in svn, for now [18:00:04] where [18:00:06] ? [18:00:08] wait. [18:00:10] petan: i got juandev to help [18:00:16] translating wikimania banners to czech [18:00:17] is your code already in git? [18:00:26] no the code is in git [18:00:31] i mean yes [18:00:35] when i'm not so busy, i'd love to do more in labs :) [18:00:39] ah, then put the debian directory in with that [18:00:58] ok [18:01:18] including the deb package? [18:01:27] i even signed it with gpg key :D [18:02:07] heh [18:02:13] not the package, plase [18:02:16] *please [18:02:20] just the debian directory [18:02:37] no need to sign the package, we do that when we put it into the repo [18:04:17] man, hydriz is making a ton of big instances in incubator [18:04:19] for bots [18:04:23] what the hell are those bots doing? [18:47:14] !log incubator suspended all incubator instances [18:47:27] wm-bot: poke poke [18:47:37] petan: is the bot dead? :( [18:55:09] 02/01/2012 - 18:55:09 - Updating keys for hydriz [18:55:12] 02/01/2012 - 18:55:11 - Updating keys for hydriz [18:55:21] 02/01/2012 - 18:55:20 - Updating keys for hydriz [18:57:40] !access [18:57:40] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [18:57:56] hm. where's morebots located [18:58:23] Ryan_Lane: I had the same question when it was down :P [18:58:30] heh [18:59:10] finding it and restarting it [18:59:30] bots-2 is looking pretty dead [18:59:58] I'm betting it's on bots-2 [19:00:10] oh. it's up [19:00:12] Ryan_Lane: Yea, cause Beetstra moved their bot off bots-2, it was killing it [19:00:54] Ryan_Lane .. i am running a beast on bots-2 .. sorry :-) [19:01:11] it's ok. [19:01:12] It is pretty much maxed out most of the time [19:01:21] I wonder where morebots is [19:01:32] 208.80.153.192 [19:01:53] ah [19:01:54] adminbot [19:02:25] \o/ [19:02:26] Ryan_Lane, LiWa3 is doing well on bots-2, it keeps up with the workload it has, hardly any lag [19:02:47] You can be proud of bots-2 .. [19:02:57] !log suspended all instances in incubator project [19:02:58] suspended is not a valid project. [19:03:11] !log incubator suspended all instances - talk to me [19:03:12] Logged the message, Master [19:03:42] Ryan_Lane: Is it really the case that you don't/can't specify a max size when you create a gluster volume? [19:03:56] I think that's the case, yea [19:03:58] *yeah [19:04:02] but you can set a quota [19:04:34] http://www.gluster.org/community/documentation/index.php/Gluster_3.2:_Managing_Directory_Quota [19:04:43] either at the directory or volume level [19:04:59] ok, that'll probably get me what I need. thanks. [19:05:13] cool. yw [19:13:09] 02/01/2012 - 19:13:09 - Updating keys for hydriz [19:13:12] 02/01/2012 - 19:13:11 - Updating keys for hydriz [19:13:21] 02/01/2012 - 19:13:20 - Updating keys for hydriz [19:49:33] Ryan_Lane: ping re "Re: Sandbox server for Wikiversity community" [19:50:25] ah. crap. did I not send them a reply? [19:50:51] * Ryan_Lane shakes fist in air [19:56:58] Ryan_Lane: looks like you didn't; you didn't cc me in any case [19:57:17] I had a draft and never sent it [20:17:10] 02/01/2012 - 20:17:09 - Updating keys for hydriz [20:17:12] 02/01/2012 - 20:17:12 - Updating keys for hydriz [20:17:20] 02/01/2012 - 20:17:20 - Updating keys for hydriz [20:21:37] Ryan_Lane: received Wikiversity mail. :) [20:21:49] heh [20:26:36] Ryan_Lane: yes? [20:26:40] bot [20:26:49] sorry, wrong bot. heh [20:27:15] ah [20:27:19] where is it located heh [20:27:22] I didn't know as well [20:27:32] !morebots is ? [20:27:33] Key exist! [20:27:36] !morebots [20:27:36] source code http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/adminlogbot/ [20:27:40] hmm [20:27:46] it's on bots-2 [20:27:48] !accountreq [20:27:48] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [20:27:49] ah [20:27:50] there's an initscript [20:27:56] ok [20:27:57] called adminbot [20:28:03] right [20:28:25] hexmode: I fixed the beta [20:28:30] it shouldn't go to oom [20:28:51] Ryan_Lane: btw is it ok to create m1.large :D [20:28:58] because hy*driz made one [20:29:09] it's fine to make them, yea [20:29:12] ok [20:29:24] but… be judicious with it ;) [20:29:30] Ryan_Lane: I'm with jerith and am working on the USSD vumi-wikipedia project. Can you provide me with access to the vumi labs machine? [20:29:31] don't make one unless you really need it [20:29:38] it eats a ton of memory [20:29:49] !account-questions | sdehaan [20:29:49] sdehaan: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [20:29:51] btw what about the privacy issue [20:30:02] did you handle that with other people in wmf [20:30:02] you'll need to send an id [20:30:06] ok, I did [20:30:12] but what about the others [20:30:16] we're having a meeting about it [20:30:19] it's going to be a PITA [20:30:20] ok [20:30:37] maybe it isn't a bad idea, though [20:30:38] let me know about the result I would need to handle that with other people in project [20:30:46] * Ryan_Lane nods [20:30:54] it took me 10 minutes to identify [20:30:54] well, it may be that we need to do this with every single labs user [20:31:00] I just scanned my id [20:31:08] indeed [20:31:17] and that is going to suck [20:31:19] philipe is going to have a lot of work [20:31:36] maybe it should be limited only to public projects [20:31:47] once you have a public dns [20:31:55] etc [20:31:58] not in private [20:32:02] instances [20:32:04] ah. could be [20:32:13] Ryan_Lane: here are my details for account-questions: https://gist.github.com/57d8266b43e12398f76e [20:33:13] you don't already have svn access, right? [20:33:34] Ryan_Lane: no [20:33:44] ok [20:35:17] !initial-login sdehaan [20:35:17] https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [20:35:19] I used Smn [20:35:27] Ryan_Lane: thanks [20:35:32] yw [20:36:21] !log bastion temporarily removing Hydriz [20:36:23] Logged the message, Master [20:39:12] 02/01/2012 - 20:39:12 - Creating a home directory for smn at /export/home/bastion/smn [20:39:23] huh what's wrong with hydri z? [20:39:45] issues with incubator [20:39:49] ah [20:39:53] load? [20:40:00] what is that project about [20:40:05] no. other issues. going to talk with him about it :) [20:40:09] ok [20:40:12] 02/01/2012 - 20:40:12 - Updating keys for smn [20:41:48] incubator project is supposed to be for working on incubator related projects [20:42:34] ah, I don't know much about it [20:42:44] I thought it's about new wikis [20:42:54] that could be merged with beta maybe [20:42:56] yeah, that's the goal of incubator [20:43:16] well, I think it's supposed to be software development related to incubator [20:43:20] ah [20:44:00] Ryan_Lane: thanks, that worked. got to bastion but I'm denied access to vumi-gw1. preilly can you help with this? [20:45:06] ah, yeah. project members need to add you to projects [20:45:23] that way they know who is being added and doing what [20:45:33] Ryan_Lane: thanks. [20:45:35] sdehaan: what is your account? [20:45:39] yw [20:45:44] preilly: smn [20:46:05] 02/01/2012 - 20:46:05 - Creating a home directory for smn at /export/home/mobile-sms/smn [20:46:34] sdehaan: okay, you should be good now [20:47:05] 02/01/2012 - 20:47:05 - Updating keys for smn [20:47:11] preilly: thanks, am in. [20:47:50] sdehaan: sweet — What are you going to be working on Simon? [20:48:45] preilly: I'm the engineering lead for praekelt, it's mostly to make sure I've got jerith's back if he's unable to work on the WP stuff. I did the original USSD proof of concept & jerith took it from there. [20:49:02] sdehaan: ah, okay — cool [20:49:32] preilly: He's my boss, no matter how vehemently he denies it. :-) [20:50:43] jerith: ha ha [20:51:08] hrm, thanks jerith [20:51:55] (He's also the best boss I've ever had. He lets me play with Wikipedia apps on company time.) [21:00:15] ;) [21:10:15] maplebed: Last week you helped me set up a proxy so I could browse a labs machine... I can still browse that one, but now want to access a new machine and can't make that work. [21:10:22] Was there a step on the actual web host that I'm forgetting? [21:10:32] nope. [21:10:35] just your laptop. [21:10:53] huh [21:12:06] so if I have proxying set up right for one labs machine I should have it set up for all of them. I must be doing something dumb. [21:12:42] the proxying was set up to proxy depending on the URL [21:12:49] so everything .wmflabs should work or something ilke that. [21:15:54] oh, it's probably the security rules. [21:16:11] Ryan_Lane, is there a simple way to add the 'web' security group to the gluster project? [21:17:20] on an instance that already exists? [21:17:38] you can go to manage security groups to add a new group to the gluster project [21:17:50] why does the gluster project need web, though? [21:17:57] oh. api [21:17:58] right [21:18:20] As long as you don't mind me tinkering with that project, I will tinker. [21:18:35] I'll need to recreate my instance? [21:23:44] if you want to add the web security group to it [21:23:45] oh [21:23:55] there's something else you could do [21:24:14] you could add the rule to default [22:11:40] Ryan_Lane: Btw I was meaning to ask - is there a way we can pull our user/pass for the nova api to interact with it directly or is that on the todo list after we get rid of the forced mw hooks? [22:12:11] Damianz: todo, we do some things that aren't supported in nova ye [22:12:13] *ter [22:12:14] err [22:12:16] *yet [22:13:07] Cooli, I assume we're waiting for puppet/dns and possible mw page stuff still. [22:28:41] Damianz: DNS, puppet, and MW [22:28:46] we can likely live without MW [22:28:51] I could write a bot for that [22:29:45] apparently there's a search in nova to find recently changed instances [22:30:00] so a bot wouldn't be amazingly terrible. heh [22:30:12] there may also be a notification service to register with [22:34:41] It would be cool if you could register hooks for events then just use the mw api. [22:43:25] Damianz: yep. it would make things *way* easier [22:44:43] Mmmmm 12year old Islay Single Malt Whiskey. I like christmas again .