[00:02:19] that could be interesting and terrifying ;) [00:02:31] Ryan_Lane: first or second? [00:02:40] both [00:02:56] Ryan_Lane: second would have enough access controls to make sure that it won't starve things (run quries on the grid, make sure they have a timeout, etc) [00:03:03] Ryan_Lane: and the former would require an ssh tunnel [00:03:11] ah, ok [00:03:21] that's cool, then [00:03:46] Ryan_Lane: yeah [00:03:56] Ryan_Lane: the second one will be open to the public, sans registration [00:04:38] Ryan_Lane: and I think that's okay, unless we have a data leak again :D [00:04:44] :D [00:05:12] Ryan_Lane: I also intend on having it follow a jsfiddle like patter, where you can share your results + the query to othere people, and they can create new queries that are modifications of yours [00:06:02] considering using a new language for it, like... scala [00:06:07] or Go [00:06:14] I'd say Ruby, but then even I'll have to laugh [00:06:26] :D [00:06:51] go or node seem to be the fashionable choices right now [00:07:32] yeah [00:07:46] if I pick scala, i won't be able to use the grid as easily [00:07:52] since fucking JVM eats all the goddamn memory [00:07:58] so I'll have to manage memory / timeouts myself [00:20:48] Ryan_Lane: Coren thoughts on giving tools subdomain names, via the proxy? will improve cookie privacy, also looks cooler :D [00:22:35] I actually discussed cookie privacy with Coren before [00:22:59] oh [00:22:59] seems as long as you don't modify the cookie policy, url space also restricts the policy [00:23:00] and? [00:23:08] I think it's safer to use subdomains, personally [00:23:14] I also think subdomains are more flexible [00:23:31] yeah, me too [00:23:40] we can also easily ssl them, with a *.tools.wmflabs.org thingy [00:23:45] thingy = ssl cert [00:24:28] yep [00:24:53] I also find subdomains much easier to reason about [00:24:57] than urlprefixes [00:25:47] in total agreement here :) [00:26:24] it still needs the proxy to support url prefixes [00:26:26] mmm [00:27:15] Ryan_Lane: actually, no. can't we currently just rewrite $1.tools.wmflabs.org/ to tools.wmflabs.org/$1/ and have them just work? [00:27:44] I don't really mind either way, but I would very much prefer that both work. [00:28:08] Coren: sure, backwards compatibility, etc [00:28:45] well, I'd send redirects from the urls to the subdomains [00:28:58] yeah that's what I was thinking too [00:29:05] mmm [00:29:21] Coren: how does the newweb thing with lighthttpd handle proxying? [00:29:24] YuviPanda: You can't make that automatic, only elective. [00:29:28] why not? [00:29:52] YuviPanda: Because there are tools that interact with each other and that'd break if you change the server name (XSS protection) [00:30:13] hmmm [00:30:55] YuviPanda: The newweb stuff is just proxied to by the frontline apache according to the rewritemap in /data/project/.system/dynamic [00:31:05] (That map is maintained by the server start/stop stuff) [00:31:10] Coren: ah, so when a new server starts up, it updates the rewritemap [00:31:11] okay [00:31:45] Coren: how are permissions handled? is /data/project/.system/dynamic writeble by all users? [00:34:37] YuviPanda: No, there is a daemon that handles that. You connect to it (unix socket), it gives you a port number and will keep it allocated as long as you keep the fd open. The lighttpd startup scrips connects, reads the port number, then forks the lighttpd daemon on that port. When the lighttpd dies, the socket closes and the daemon removes the entry. [00:34:50] :D [00:35:04] right, almost the same as I'd need for the nginx based proxy [00:35:09] Coren: where's the code for this deamon [00:35:54] In puppet. modules/toollabs/files/portgranter [00:36:06] sweet [00:36:08] modules/toollabs/files/portgrabber is the counterpart [00:36:24] alright! [00:36:30] i'm going to go try grab some sleep now [00:36:40] and see if I can reset this sleep cycle again [00:36:44] cya! [00:36:47] and thanks Coren / Ryan_Lane :) [00:36:49] * Coren waves. [00:55:03] TParis: Sorry, was afk for ages waiting for towtruck. [00:55:24] Is http://utrs.wmflabs.org/privacy.php a policy specific to utrs or is it just the labs privacy policy recapitulated? [01:28:02] andrewbogott: It was the WMF Legal-semi-approved privacy policy on toolserver [01:28:45] It was "We approved it but we're not giving you legal advice" approved. [01:40:19] TParis: OK :) It would be good for you to add '…and the wikimedia labs Terms of Service' with a link someplace on that front page. [02:05:19] okay Andy [02:05:29] Do you think I need to get an explicit agreement from the UTRS users as well? [02:05:47] Their private data isnt recorded but some of them have access to the data of the appeals [04:05:15] andrewbogott: Are you about? [04:07:04] YuviPanda|away? [04:16:30] hey TParis [04:16:34] sorry was away [04:17:15] TParis: 'sup? [04:18:01] YuviPanda: he wanted to know about getting a SSL cert [04:18:11] ah [04:18:38] needs to talk to Ryan_Lane about it [04:18:48] and might have to definitely sign an NDA [04:18:59] How do I save a file in emacs23? [04:19:09] why is an NDA needed? TS never required one with an SSL cert [04:19:12] TParis: c-x c-s [04:19:41] legoktm: I'm not exactly sure, but because he'll have access to the root cert? [04:19:54] legoktm: toollabs gives you ssl without an NDA [04:19:55] hrm. [04:20:07] legoktm: and if you don't need user host IP, the dynamicproxy gives you ssl for free without NDA [04:20:08] yeah [04:20:46] All dev's would need to sign it then [04:20:52] or sign one* [04:20:56] TParis: rather, everyone with root [04:21:14] hmm, or I could make the proxy sometimes set X-Forwarded-For headers [04:21:16] Yuvi: I have UTRS working, is it possible for you to take away root for all of us for now and install the vert? [04:21:25] cert* [04:23:01] how do I quit emacs23? [04:23:05] never using it again [04:23:12] TParis: tbh, I'm not sure how to do it. [04:23:29] TParis: install a specific certificate on one machine only [04:23:35] TParis: I don't even know if I can do it [04:23:38] Ryan_Lane: around? [04:23:39] EXPORT $editor=nano [04:24:37] we don't give out real certs to folks who don't have NDAs [04:24:54] Ryan_Lane: I am considering just setting XFF from the proxy [04:25:01] that's an option [04:25:02] Ryan_Lane: it'll mimic current usage patterns of people with public IP [04:25:06] but it's manually set, right? [04:25:18] if we can do that, then UTRS isn't actually necessary right? [04:25:20] the project [04:25:27] we could probably do the same in tools [04:25:31] Ryan_Lane: huh? [04:25:36] Ryan_Lane: tools has a stronger privacy policy [04:25:39] no access to XFF :P [04:25:42] or user IP [04:25:45] ah. right [04:25:48] this will have a different PP [04:25:52] yeah, then via yuviproxy it is [04:25:58] Plus, I've already launched and emailed all of my users [04:25:58] Ryan_Lane: okay, let me make the XFF patch then [04:26:06] is there any non-manual way to do this? [04:26:16] TParis: heh. wasn't that a little premature? :) [04:26:35] No, the ToS don't require me to have SSL from my reading. But some folks suggested it. [04:26:45] TParis: I too suggest it, SSL everywhere == good [04:26:57] well, ideally you'd have protocol relative links to this [04:26:57] Ryan_Lane: so I'm going to modify the proxy to pass XFF for everyone [04:27:03] Yeah, fair enough, but what about just removing all of us w/o NDAs from the root until I can sign an NDA? [04:27:08] YuviPanda: ugh. can you make it optional? [04:27:18] Ryan_Lane: well, where do you store that as an option? [04:27:19] TParis: we'd still need to buy a cert [04:27:25] ic [04:27:42] Ryan_Lane: we can have two proxy machines, one which XFFs and one which doesn't [04:27:57] how would you select that from the proxy window? :) [04:28:10] why can't XFF be an option stored in redis? [04:28:19] Ryan_Lane: hmm, it can... [04:28:29] there's no *.wmflabs.org cert? [04:28:39] that normal people have access to? :) [04:28:40] legoktm: there is, that's what the proxy uses [04:28:40] no [04:28:56] TParis: is it okay if it takes a while (~week) to get you SSL? [04:32:12] YuviPanda: That's fine, the toolserver version only got SSL recently itself [04:32:24] ah, alright [04:32:31] TParis: can you file a bug so that I don't forget? [04:32:36] If you email me the NDA, I'll get everyone to sign it [04:32:40] yes [04:33:20] TParis: if I do it via the proxy, no NDA needed [04:33:33] And judging from Ryan_Lane's responses, I think doing it via proxy is the preferred approach here [04:34:27] Okay, is that going to change anything about UTRS? Like the URL or anything? [04:36:22] TParis: nope [04:36:37] TParis: hmm, will change the public IP, and you will have to login via bastion. [04:36:45] no user facing changes other than that [04:36:57] url will stay the same [04:37:04] damn, was very handy not using bastion [04:37:26] heh, once you get the setup done, it's equally trivial using bastion [04:43:30] Hmm, I cant find the php install on the server [04:43:33] which php doesnt work [04:43:51] TParis: let me check [04:45:06] TParis: fixed [04:45:11] php5-cli wasn't installed [04:45:23] now let me add that to the puppet config [04:45:34] TParis: either way, you should be able to use php from the commandline now [04:45:40] thanks [04:45:42] TParis: w [04:45:44] yw [04:45:58] TParis: if you need to log into any instances other than the web you'd need to use a bastion of some sort anyway [04:46:57] When I try to run "become calling-card" (for a new project I created), I get "sudo: sorry, a password is required to run sudo". I don't remember setting a password .... or know how to set one. [04:47:17] lfaraone: log out and log back in [04:47:19] should work [05:02:39] alrighty, the private data removal script works [05:02:44] I think that's it, migration complete [05:03:09] TParis: woo! congrats on the migration :) [05:03:34] thanks, and thanks again for your help. Once SSL is set up, I'll be completely done (aka, no one will bug me) [05:03:59] I also set up a new script that forces our already registered users to agree to the WMF Labs ToS [05:05:29] TParis: :D [05:07:36] wish someone would create an appeal so I can get passed developer's bias [05:10:21] gnight [05:11:54] TParis: night [10:27:25] If I have a Tool Labs account, is "krd@wmflabs.org" the e-mail address I have there? [12:15:40] is something wrong with the webserver? [12:15:57] fist requests took ages, then one timed out with this: [12:15:58] Proxy Error The proxy server received an invalid response from an upstream server. [12:15:59] The proxy server could not handle the request GET /render/tlgbe/tlgwsgi.py. Reason: Error reading from remote server [12:16:04] *first [13:57:01] Coren, I am now certain that something about PHP on labs is misconfigured. I'm getting hundreds of unserialization errors with afdbot.php where I don't get a single error when run continuously on my computer. [13:58:29] Other scripts that have been in operation are also throwing unserialization errors. [13:58:53] RfX-tally and cratstats are randomly failing, when they used to run just fine. [14:00:03] Coren: puppets not running right [14:00:48] Betacommand: "puppets"? [14:01:19] Cyberpower678: Labs uses the same php as production; I'll need a bit more detail than that to diagnose. [14:01:58] Betacommand: Ah, puppet runs. Yes, I see someone has broken something over the weekend. [14:02:16] Have a look at afdbot.err located in /data/project/cyberbot/CyberbotI [14:02:19] Coren, ^ [14:03:38] You may also want to look at afdbot.out [14:10:29] If I have a Tool Labs account, do I have an email addresse attached to this account? If yes, at which host/domain? [14:22:51] Coren, have you had a look? [14:24:08] Cyberpower678: This doesn't contain anywhere near enough information to debug anything. [14:25:01] Coren, how about this? These malfunctions have been arising since you guys updated PHP. [14:30:17] Coren, ? [14:33:01] Cyberpower678: Excuse me, but do you expect me to be able to diagnose the issue with 10 minutes of cursory looking? [14:33:33] Coren, no. But you weren't responding, so I was testing to see if you were away. [14:38:49] Cyberpower678: Wait, you're using serialize to move data between different web servers? [14:39:16] ??? Now I need more data. [14:39:40] $data = unserialize( $this->get_http()->get($this->base_url, $arrayParams)); [14:40:03] It's retreiving serialized data from the API. [14:40:21] The API serializes the data. [14:40:45] But that's not the issue, since it works fine on my computer. [14:41:10] And that has always worked on labs until October. [14:45:46] Coren, if I'm not online when you want to respond, memoserv me. [14:46:08] The error is always unfailingly at the beginning of the serialized string; this doesn't look like an unserialization error but a data error. Please log the actual data being unserialized so we can figure it out. [14:47:20] Alright. I'll have Peachy log and mark it if it failed to unserialize. [14:47:25] Bye for now. [15:30:21] petan? [15:30:46] Steinsplitter? [15:32:12] see pm *augenroll* [16:17:51] Coren: can you give any more info on this? http://lists.wikimedia.org/pipermail/labs-l/2013-October/001766.html [16:18:03] is it planned to have those DBs available? [16:18:36] Hm, yes actually -- I'm done testing them and should deploy now. [16:19:02] * Coren puts this on his TODO for the week. [16:19:14] cool [16:19:58] that would be great Coren, thx [17:04:10] Coren, hi. [18:53:50] Coren: any way to trigger an alert or email when a sudo apt-get install is done manually? :D [18:55:39] we could exclude it in sudo rules [18:56:19] well I'm okay with doing a sudo apt-get install as long as there's a puppet patch right after it [19:01:46] Change on 12mediawiki a page Wikimedia Labs/Tool Labs was modified, changed by 38.104.158.106 link https://www.mediawiki.org/w/index.php?diff=818899 edit summary: /* Goals */ [19:02:32] Change on 12mediawiki a page Wikimedia Labs/Tool Labs was modified, changed by 217.251.192.14 link https://www.mediawiki.org/w/index.php?diff=818900 edit summary: Undo revision 818899 by [[Special:Contributions/38.104.158.106|38.104.158.106]] ([[User talk:38.104.158.106|talk]]) vandalism [19:11:42] YuviPanda: wouldn't it make more sense to force-run puppet with the patch that's submitted-but-maybe-not-reviewed? [19:11:56] valhallasw: we can't do that, sadly [19:12:03] hence the terrible alternative [19:12:25] huh, why not? [19:12:46] git clone puppet-repository && git review -d 12345 && puppet something.pp? [19:12:49] valhallasw: the puppetmaster - where all the machines get their config from, tracks master [19:12:54] valhallasw: nope! not in our setup [19:13:01] oh, there is no local puppet? [19:13:12] valhallasw: nope [19:13:17] there's a way to get local puppet [19:13:18] byt [19:13:30] but it is to be used only for testing puppet changes themselves [19:13:39] none of the actual instances used havre thata [19:21:00] YuviPanda: sorry, that last sentence didn't make sense [19:21:11] gah [19:21:12] i [19:21:14] i mean [19:21:16] none [19:21:18] gah [19:21:19] new [19:21:21] keyboard [19:21:24] :D [19:21:28] keep mixing up keys [19:21:35] anyway, it is called self puppetmaster [19:21:43] and none of the instances in toollabs use that [19:21:43] one of the split ones to prevent RSI? [19:21:45] YuviPanda, new keyboard as in Chinese keyboard? [19:21:57] so we can't run it by applying the patch [19:22:02] Right. [19:22:11] yeah, the kinesis advantage pro [19:22:18] really gets a while to get used to [19:22:59] YuviPanda, I'll take it, if you can't get used to it. Even my parrot has a better hang of the keyboard. :p [19:23:01] :D [19:23:13] :) [19:23:35] YuviPanda: I feel your pain. I also have one, but I mainly use other keyboard, so it takes a while getting used to every time I use it... [19:23:45] heh yeah [19:23:57] re learning all the keys almost [19:24:08] I've got a keyless keyboard. :p [19:26:21] http://xkcd.com/1284/ [19:26:50] I saw that one. [19:27:14] The trick to those keyboards is to use your voice. [20:14:06] andrewbogott: would you be the person to ping to create a project (when you have a free moment)? -- https://wikitech.wikimedia.org/wiki/New_Project_Request/collection-alt-renderer [20:14:25] mwalker: yep, just give me a few minutes... [20:14:36] awesome; thanks most kindly :) [20:21:02] mwalker: ok, created -- do you need anything else? [20:21:11] not that I know of right now [20:22:43] cool -- let me know if things aren't working properly. [21:37:35] legoktm: what's external load balancer mean? [21:38:53] they use a different database which is hosted on a different load balancer as I understand it [21:39:11] https://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php <-- search for 'wmgEchoCluster' [21:41:46] i don't think any DBs have load balancers? [21:42:11] that's apparently just a reference to https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php [21:42:16] wow, internets so slow... [21:43:16] / Use the main db if this is set to false, to use a specific external db, just [21:43:16] / use any key defined in $wgExternalServers [21:43:16] $wgEchoCluster = false; [21:43:39] maybe load balancer was the wrong word [21:44:54] https://github.com/wikimedia/mediawiki-extensions-Echo/blob/master/includes/EchoDbFactory.php#L22 says "getExternalLB" [23:30:58] MaxSem: I'm guessing you're asleep? [23:31:07] no [23:31:10] just busy