[00:18:15] andrewbogott_afk: [00:18:17] gah [00:18:19] nvm [00:23:39] horrific question... I have a local development machine that needs to support two way communication with a labs instance -- so I'm trying to set up tunnels for that [00:24:19] but it seems that a ssh RemoteForward cannot talk to a different ssh sessions LocalForward [00:24:45] anyone know how to do this? [00:30:44] mwalker: why not just create two labs instances? [00:31:14] mostly because I already have my local instance already setup [00:32:27] I don't think this is a simple problem [00:32:35] no; it's not :( [00:34:17] ... just going ahead and setting up a new labs instance and I'll just forgo my nice local features [00:48:11] Ryan_Lane: is it possible to change the security group of an instance after it has been created? [00:49:30] you can't add/remove security groups after an instance is created [00:49:33] it's annoying, yes [00:49:59] *sadness* [00:50:01] we should probably add that feature into openstack at some point [00:50:13] can I manually punch iptables in the face? [00:50:17] heh [00:50:29] hm, I wonder if neutron already has this done [00:50:32] I'm betting not [00:51:32] * mwalker *blinks* iptables is not installed in the default image [00:51:42] how are we applying firewall rules? [00:51:53] is that an openstack virtual network thing? [00:51:59] that's what security groups do [00:52:11] iptables isn't installed? :D [00:52:15] well, that's a bug [00:52:25] not on this 12.04 instance I just spun up [00:52:27] creating images from scratch is a pain in the ass [00:52:35] it's missing man, too [00:53:30] ah [00:53:36] I bet I need the ubuntu-standard package [00:54:03] yep, that's it [00:56:52] fucking ubuntu-standard adds popularity-contest [00:56:56] I hate ubuntu [00:57:11] yep [00:57:24] I'm not sure there is any distro I do like though [00:57:37] I think debian is a better distro than most [00:57:50] openstack would be more difficult to use on debian, though [00:58:13] because it needs things that aren't in unstable? [00:58:39] because ubuntu has the "cloud-archive": https://wiki.ubuntu.com/ServerTeam/CloudArchive [01:22:04] who runs apt.wikimedia.org? [01:22:17] I'd like to tweak one of the builds, php5-redis [01:59:50] Ryan_Lane / YuviPanda: Is it possible to copy/clone an instance? [02:00:15] not really no [02:00:32] well poo [02:00:34] Thanks [02:00:40] TParis: what is the use case? [02:00:56] in general we recommend puppet for this use case [02:01:03] dev server - mirror the environment [02:01:38] Ryan_Lane: I find it hard to reccomend puppet to people who just want to run stuff - like TParis, considering how hard it is to get them merged and also our repos are a mess anyway [02:01:46] might be easier when we have per project puppet [02:01:55] if they don't use puppet and their instance just goes away, they're fucked [02:02:07] the motto of cloud services is "cattle, not pets" [02:02:18] YuviPanda: Honestly, it'd be easier if you have a [[Help:Making your first server]] which step by step click here and there and type this directions.... [02:02:35] The help documents seem to be tailored to users with at least some cloud experience. [02:02:37] I'm not saying that 'hand build everything!' is what we should do, Ryan_Lane [02:02:45] your instance is not a pet, we make no guarantees it'll live [02:02:56] it could disappear at any time [02:02:59] Ryan_Lane: just that ops/puppet.git as it stands isn't really a viable solution and we need to look for alternatives [02:03:03] why not? [02:03:08] make a module, push it in [02:03:10] we'll review it [02:03:22] hell, you could have your own repo with its own modules [02:03:28] and then use puppetmaster self [02:03:35] indeed that's way better as a solution [02:03:43] than putting everything in ops/puppet [02:03:57] i should've initially said that I find it hard to reccomend ops/puppet [02:04:00] not puppet [02:04:02] in the future you'll just be able to run puppet with your module in the includes path [02:04:02] puppet is fun [02:04:10] that won't work until everything is modules though [02:04:25] in the future, yeah [02:04:28] not right now tho [02:04:37] TParis: heh, true. [02:04:43] actually, it's possible to write your own module and use puppet apply with the module [02:04:50] right now [02:04:59] indeed, or just have your local repo [02:05:02] assuming it doesn't need to reference anything from the production repo [02:05:03] like labs-vagrant [02:05:16] but there isn't any docs, etc on it yet [02:05:32] and that's also why I explicitly mentioned 'people like TParis'. [02:05:34] anyway, yeah, it's best to assume your instance could die at any point [02:05:36] :) [02:05:41] oh well. small enough population [02:05:50] Ryan_Lane: yeah, I told him that too [02:06:01] Ryan_Lane: backed up via cron to NFS [02:06:06] seems to be a stopgap fix [02:06:23] well, that's a legitimate way to handle your data [02:06:29] config less so, but it works [02:06:43] Ryan_Lane: the only config in this case is the lamp role [02:06:46] so I guess it's fine [02:06:50] * Ryan_Lane nods [02:07:01] yep, that should be fine [02:07:21] I actually recommend accessing your data in the local instance and backing it up to NFS [02:07:26] assuming the set of data is small [02:07:34] that way you have two copies of the data [02:07:43] TParis: anyway back to your original question - no you have to set that up yourself [02:07:51] which should be easy enough with git [02:07:52] hahaha [02:07:54] you guys are funny [02:08:11] for some definition of 'easy enough' [02:08:15] Well that's okay, at least it's working for now, the rest can come to that point. [02:08:19] :D [02:08:25] Once Crazycomputers gets on board with the new server, he'll work something out. [02:08:46] You guys are smart, but I think Crazycomputers puts everyone to shame. He's literally crazy about computers. [02:09:04] That's not meant to be an insult...btw...not sure how it came accross [02:09:22] hehe [02:09:53] Ryan_Lane: I should spend some time putting up docs about generalizing how labs-vagrant is run [02:10:13] Ryan_Lane: I think there's good value in that and will also encourage more people to use puppet if it is something they can +2 themselves [02:11:17] Quick check, there is no problem with me restarting the instance since I'm the only one on it, right? [02:11:52] yeah feel free to [03:11:50] TParis, I missed you last night -- did you still have questions or are you sorted? [03:13:26] hey AndorraLaVella_ [03:13:31] gah [03:13:33] andrewbogott: [03:13:50] * andrewbogott waves [03:14:03] * andrewbogott is healous of AndorraLaVella_'s name [03:14:07] *jealous [03:14:24] andrewbogott: can you create a gerrit repo for me or is it too late in your tz? [03:14:37] not urgent, can wait a day or two, but... [03:14:53] I can, if I remember how... [03:14:57] hehe [03:15:19] What do you want me to call it? And do you want the 'initial empty commit' or are you going to do an import? [03:15:20] andrewbogott: meh, forget it. I can't really push anything there today anyway. [03:15:34] ok, well, let me know :) [03:15:43] andrewbogott: thanks! and sorry for the noise [05:35:00] !log logstash Got POC install running. See [[mw:User:BDavis_(WMF)/Projects/Logstash]] [05:35:03] Logged the message, Master [12:46:54] Hi there. Repeating question from yesterday: When I send out an e-mail from tools-login by mutt, it goes out from krd@wmflabs.org but wmflabs.org doesn't have an MX entry and doesn't listen at 25/tcp. What is wrong? [15:25:22] Coren, I owe you an apology. It's not a labs issue. It's a Wikipedia issue. [15:25:43] No apology necessary, that's why I said you'd want to log the actual data. :-) [15:26:36] Well. Whenever I attempt to edit, I get the technical issues banner returned. That's why PHP can't unserialize the data. [15:26:41] krd: We're still working on mail relay. But your outgoing mail /should/ be saying krd@tools.wmflabs.org not krd@wmflabs.org it's a bug. [15:27:13] Coren, It seems to only work 10% of the time. [15:27:22] Cyberpower678: That sounds like a serious bug; have you reported it? [15:27:50] Not yet. I only just now got this logged data. Would you like a copy? [15:28:28] I don't, but you should certainly include excerpts of it in your bug report. [15:29:10] I'll make sure Faidon's attention is on it; I know he and Mark have been doing some changes in caching that may be related and they'd want to know about it. [15:29:28] Coren, Peachy is designed to make 20 consecutive attempts before terminating. Scripts keep terminating on labs and I started digging deeper. All of my scripts are being affected.. [15:30:35] what's up? [15:30:41] (I'm Faidon) [15:30:42] Hm. Strangely enough, not everything broke and there are no hordes beating down our doors about not being able to edit. That either means peachy is doing things in a slightly different way that confuse matters, or that the caches you hit from inside labs have problems not shared by most others making it rare in general. [15:31:44] paravoid: A bot doing edit from the labs is getting frequent errors on edits, and it only started doing it recently. [15:32:29] hm [15:32:41] can I get the output of such an error message? [15:32:49] Cyberpower678: ^^ [15:33:00] I'm looking to find out if it's an error from apaches or varnish [15:33:15] we've been dealing with all kinds of issues these past two weeks :( [15:33:50] (but don't assume it's something we know already, let's get to the bottom of this) [15:34:23] This one at least should be fairly easy to reproduce. Cyberpower678 says it's intermittent but has about 90% hit rate. (I suppose it might be better said that it /works/ intermittently) [15:34:37] Coren: Peachy's API engine hasn't changed since 2010. It's been working until recently. [15:35:02] CP678|iPhone: Can you give paravoid the actual error? [15:36:08] paravoid: I'm on my phone right now. Send me an email through Wikipedia, and I'll reply back with the log data. It's something about Varnish apparently. [15:38:06] Coren: ^^ [15:39:53] Coren: Combined with the problem of wrong permissions of my homedir from yesterday, is it possible that my account has not been created correctly? [15:40:42] krd: It's possible; what seems to be the issue with it? [15:41:08] Well, the outgoing-mail problem i mentioned. [15:42:31] Got to go for now. [15:43:16] krd: Hm, no, that should be unrelated. I'll look into it. But you wouldn't be able to receive email anyways -- that should arrive in a week or two now that all the legal hurdles are passed. [15:43:37] Coren: Ok. [17:42:04] !add-labs-user [17:42:56] :( [17:43:09] Does anyone recall the process (or have a doc link) for how to link a labs account to an existing SVN user? [17:43:21] Coren, I sent you an email/ [17:43:34] Yes I saw. [17:44:01] andrewbogott: No on either count; I don't recall it having been done since I'm here. [17:44:20] There was a doc page for it, but I fear it's been 'cleaned up' since that almost never happens [17:44:21] It might have, but Ryan wouldn't have told me. [17:44:49] I thought all SVN users were imported? [17:45:34] Coren: https://bugzilla.wikimedia.org/show_bug.cgi?id=56777 [17:45:41] I would've thought so as well... [17:45:46] could be that there's some other bug they're hitting [17:46:44] andrewbogott: shinjiman:x:1121:550:shinjiman:/home/shinjiman:/bin/bash [17:46:44] The user exists. [17:47:19] What about on wikitech? [17:47:21] * andrewbogott checks ldap [17:48:37] andrewbogott: I don't see him on wikitech with Special:ListUsers. I note that his GECOS field has a lowercase username. [17:48:53] Hm. So does mine, so it's meaningless. [17:54:59] (Sorry, phone) [17:57:58] well, I see shinjiman in ldap but there's no associated email. So I don't know how they'd do a password reset. [17:58:09] Guess Ryan will show up any minute and have all the answers. [19:10:29] why is granting shell so slow? is ldap in the same DC as OSM? [19:10:45] the group is probably getting a little big [19:11:50] that i can't dispute [19:12:13] I wonder if there's a way to handle that more efficiently in LDAP [19:12:24] it's not just today, it's a few different days i noticed this. edits (with "edit this form") are much faster than adding shell [19:12:42] the way you add to a group is painful [19:12:55] well we could automate with a user script [19:12:57] a la twinkle [19:13:13] it's not due to the interface [19:13:23] it's the way you need to add members [19:13:36] though I've noticed LDAP is having some performance issues as of late, too [19:14:20] well regardless, if you can just check some boxes and let JS work in the background and come back and look in a few mins then the perf issue is less painful [19:15:19] I'd rather solve the performance issue rather than hiding it [19:15:31] i'm not stopping you :) [19:16:33] separate issue: we didn't have a solution for mediawiki project yet, right? i had asked about making a debconfwiki project [19:16:39] wait. shell is a mediawiki group [19:16:47] it's not an LDAP group (yet) [19:16:56] so there's no reason there should be performance issues there [19:17:05] i just know what i see [19:17:07] hell, LDAP might actually be faster [19:17:27] that interface shouldn't talk to LDAP at all [19:17:35] hm, maybe it does. [19:22:51] btw, it's not necessarily shell that's slow. could be bastion [19:23:00] both are added with one click [19:24:31] oh [19:24:31] right [19:24:34] could be bastion [20:17:03] Ryan_Lane: got a few speedy questions if your around? :) [20:17:31] sure, what's up? [20:19:01] Is there somewhere to see who is in what ldap group? Who can modify the ldap groups (whats the process for this). What requirments would we need to meet at wmde to get our ldap access to graphite? :) [20:19:25] from labs, you can do: ldaplist -l group wmde [20:19:31] any labs instance [20:19:45] only ops can modify the groups right now and an RT ticket is the way to do so [20:20:07] maybe we can eventually open up group modification to others, but it isn't easy [20:20:31] -bash: /usr/local/sbin/ldaplist: Permission denied ;p [20:20:39] -_- [20:20:44] run it via python [20:20:46] that's annoying [20:20:51] can you open a bug for that? [20:21:07] wikimedia labs product, infrastructure component [20:21:21] will do :P [20:21:27] run it via python? HAH [20:22:33] Ryan_Lane: > https://bugzilla.wikimedia.org/show_bug.cgi?id=57028 [20:22:40] thanks [20:22:53] it's just missing the x bit for other [20:50:07] Cyberpower678: We're deploying changes to DNS that should avoid the ulsfo varnish caches for labs; can you try again and see if your problem goes away? [21:03:07] Coren, if the problems have gone away, it should fix itself. [21:31:21] "Requirements 5 years of professional experience in a DevOps" < Seriously? IIRC that term only really took off in popularity around 2009... and in concept it's the same as 'the dude who just figures out how to make things work', which is any good dude imo [21:41:32] Damianz: devops is just a term [21:41:47] Damianz: I have been doing something similar to devops in a previous job back in 2006-2007 [21:42:14] Damianz: yeah, it's kind of dumb [21:42:17] Damianz: think about the term as an implied list of skills looked at [21:42:26] if you've done ops or dev, just say you have devops experience [21:42:28] done [21:45:06] hashar: True, but I don't believe anyone can do ops effectivly without dev skills or design things in dev effectivly without thinking about ops... [21:45:39] * Damianz doesn't even mention the version of devops where qa is overlapped as well... since if you don't test your own stuff in dev or ops you're going to do bad things quickly [22:04:03] Coren: tools-login (and one other instance) include the class 'imagescaler' but I'm having a hard time figuring out if there ever was such a class. [22:04:10] There is role::applicationserver::imagescaler... [22:04:21] which, if I include that instead, I get some redefinition complaints [22:04:29] Any idea what's up with that? [22:05:08] andrewbogott: Giving an interview atm, be right with you when I'm done. [22:05:18] 'k sorry to interrupt [22:14:19] andrewbogott: how often does pupet run on tools labs instances? :P [22:14:41] addshore: hourly [22:14:51] but puppet is broken on tools-login. I'm working on that now, kind of down a rabbit hole [22:14:56] * addshore presumes the MOTD is just wrong then? :P [22:15:27] for tools-dev apparently the last puppet run was Thu Sep 26 :D [22:16:15] addshore: nope, that's accurate -- puppet is switched off on tools-dev [22:16:27] You'll have to take that up with… whoever it is that does such things on that instance. [22:16:34] ahh, okay :P [22:16:48] how do you switch it off? [22:17:18] i'm not sure… and, anyway, it's highly discouraged :) [22:17:44] I guess you just turn a cronjob off somewhere? :d [22:17:52] No, it's someting like puppetd -tv [22:18:03] yum! [22:18:05] It's not just not cronned, it literally refuses to run when I do 'puppetd -tv' [22:18:19] Um… wait, I"m not making sense [22:18:26] "notice: Skipping run of Puppet configuration client; administratively disabled; use 'puppet Puppet configuration client --enable' to re-enable." [22:18:43] As I recall, the command in that message is not actually correct, but you get the idea :/ [22:18:59] great [22:21:56] * Damianz reduces andrewbogott's coffee intake [22:23:00] * addshore adds sleeping pills [22:23:22] * andrewbogott doesn't drink coffee, is just naturally jittery [22:36:07] !log tools removed 'imagescaler' class from tools-login because that class hasn't existed for a year. Which, a year ago is before that instance even existed so what the heck? [22:36:09] Logged the message, dummy [22:39:36] !log editor-engagement removed 'imagescaler' class from instance 'pronunciationrecording' because that class doesn't exist. [22:39:37] Logged the message, dummy [22:45:13] How can you remove it if it doesn't exist :D [22:46:32] I removed the lack of it. [22:46:43] So you added it? :D [22:48:33] It's the difference between 0 and the empty set [22:48:46] * andrewbogott is, and is not [22:49:27] we really need to fix the puppet groups interface [22:49:58] It's horrible and manual and should be automatic and wonderful [22:50:00] there's lots of stuff in there that's totally unusable [22:50:04] yes [22:50:20] the global part can more easily be automated [22:50:28] the per-project part probably needs to stay manual [22:51:22] Imo if I branch puppet, push it up to gerrit and then want to deploy that branch I should just select it on the interface, get the environment by doing that and have the interface switch to my new classes instantly [22:51:27] That would be SEXY [22:51:40] yeah, we'd need the branch to be remote, though [22:51:46] which I'm also fine with [22:52:00] in fact, we can allow projects to make and manage branches [22:52:09] since we can base it off of the project group [22:52:18] assuming the branch is named the same as the group [23:10:07] ^demon|away: are you really away? I'm looking to get gerrit permissions right for the new QA/VE person Jeff Hall [23:10:29] <^demon|away> Yo [23:10:44] ^demon|away: permissions in gerrit I mean [23:11:00] ^demon|away: he should have +2 where I have +2 but does not seem to right now [23:11:14] <^demon|away> Probably wasn't added to wmf group, sec. [23:12:26] <^demon|away> Hmm, has he logged into gerrit yet? Can't seem to find him. [23:12:37] <^demon|away> Ah there, nvm [23:13:17] <^demon|away> {{done}} [23:13:41] thanks ^demon|away [23:13:45] <^demon|away> yw. [23:13:49] <^demon|away> Ryan_Lane: Question: when formey dies, where should I run commands like ldaplist & so forth from? [23:19:45] ^demon|away: I dunno :) [23:20:19] <^demon|away> Guess it just means I'll never do ldap again ;-) [23:20:25] :D [23:23:41] Coren: Ok, I think tools-login is sorted now. [23:23:44] (And I'm out.)