[10:54:21] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413194 (10valhallasw) [11:07:31] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413228 (10valhallasw) > Disable querry, as it goes against labs usage policy (allowing arbitrary queries to non lab users) Note that this is a Tool Labs policy, not a generic Labs policy. [11:16:04] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413264 (10yuvipanda) So the measures that Quarry has in place to prevent abuse are: # Tied to user accounts on SUL, and the queries themselves have the username attached. We can trout people wh... [11:17:56] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413266 (10yuvipanda) I'll note that the only person who seems to have run toxic queries has been you :) http://quarry.wmflabs.org/JCrespo%20(WMF) has broken Quarry as well. [11:20:27] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413276 (10yuvipanda) ('broken' in the sense that it is reporting non-running queries (they were killed too) as running, not that Quarry itself isn't working anymore) [11:20:50] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413277 (10jcrespo) My suggestions: * queries should be limited to 1 per account * There should be a throttling on new account creation [11:21:27] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413278 (10jcrespo) [11:21:52] 10Quarry: Limit number of concurrent queries one user can run at a time - https://phabricator.wikimedia.org/T104316#1413282 (10yuvipanda) 3NEW [11:23:29] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413290 (10yuvipanda) I've filed T104316 for #1. Can you explain what you mean by #2? Quarry's logins are just Wikimedia SUL accounts... I think '1' is too stringent a limit - Quarry has never c... [11:28:12] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413316 (10jcrespo) Accounts probably should be autoconfirmed, or have a verified email to get in contact with them. Or maybe have a default small query limit (1 min), make it longer for older a... [11:30:45] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413318 (10yuvipanda) I think we should be as permissible as possible and do restrictions based only on actual abuse than imagined. Autoconfirmed does not work - autoconfirmed on which wiki? meta... [11:49:13] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413354 (10jcrespo) 5Open>3Invalid [12:26:41] 10Quarry: Have an easy way to ban users from Quarry - https://phabricator.wikimedia.org/T104322#1413399 (10yuvipanda) 3NEW [12:26:52] 10Quarry, 6Labs: Quarry is a tested way to perform a DOS against labsdb - https://phabricator.wikimedia.org/T104308#1413409 (10yuvipanda) Filed T104322 as well as follow up. [13:01:02] ^ All this wikibugs might be a problem. [13:03:02] halfak: yeah, was just 'coz today was drama tho. [13:03:06] halfak: I'll be happy to get it out of here if you want. [13:03:35] Not sure. [13:19:51] halfak: let me know if you do change your mind :) [13:20:21] Yeah... It's nice to have wikibugs ping. I wonder if there is a rate-limit setting [13:20:40] E.g. ping about a single phab task at most once per hour. [13:34:31] milimetric: my offer to redo wikimetrics' puppet isn't forgotten, btw :) just still recovering from NFS failures... [13:34:53] milimetric: also, it looks like ORES might also use celery - at which point I'll just write a generic nice celery module that ORES / Quarry / Wikimetrics can share :0 [13:35:08] cool YuviPanda, I was going to send an email about that the other day but I figured you were busy [13:35:20] milimetric: you did send me an email, though? [13:35:22] cool, generic celery would be great [13:35:25] note that on the bug as well maybe [13:35:39] I did?! :) oh boy, I'm getting old sorry :) [13:35:41] :D [13:36:06] well, then I was struggling with sending it, I remember that [13:39:18] o/ milimetric [13:39:25] :) [13:39:32] hey halfak [13:39:39] oh! I forgot our meeting [13:39:40] Just got done working with joal on his new input format for doing XML dump processing. [13:39:57] milimetric: in more self hosted puppetmaster woes, I just lost a machine I forgot was a self hosted puppetmaster (android-build, was doing alpha builds for the android apps team) [13:39:59] No worries. We called it early so that joal can do some more work on it so that Hadoop streaming likes it. [13:40:05] nobody can ssh in anymore because I forgot to fiddle... [13:40:09] But I'm excited to have time for this again :) [13:40:13] * YuviPanda preaches the anti self hosted puppetmaster thing [13:40:15] sweet [13:40:21] halfak: how do you have so much energy!!!1 [13:40:23] * YuviPanda is jealous [13:40:25] :) [13:40:34] brt [13:40:56] I dunno man. Was hard to get out of bed this morning. [13:41:00] So comfortable. [13:41:04] So much sleep [13:41:49] * halfak copies XML files to altiscale HDFS :S [13:41:51] *:D [13:42:38] is that a tears of joy emoticon? [13:43:40] ;_; [13:44:54] ¯\(◉◡◔)/¯ [13:46:07] ಥ ◡ ಥ [13:46:09] There [13:46:10] Got it [13:47:17] Whoa [13:47:33] halfak, I'm concerned about your right eye being overly dilated. You might be having a stroke. [13:48:25] You could probably really troll people with some iris dilation fluid. [14:51:24] Bonjour. [14:57:33] o/ guillom [14:58:46] How goes? [14:59:03] Not bad. :) I just got time to pick up that content persistence work again. [14:59:12] I'm generating data for the most recent enwiki XML dump. [14:59:34] (re. measuring value-added & more specifically measuring edit productivity) [15:01:01] * guillom managed to get the first version of his licensing metrics script to work yesterday \o/ [15:04:28] halfak: I had missed that last bit. Nifty! [15:05:12] Any output of that licensing script you want to share at the RG meeting? [15:06:36] Not yet; For now it's just counting the number of free and unfree files on all projects. But by Wikimania, I'm hoping to have more metrics, like distribution of licenses over time, by file type, etc. [15:07:13] We use so many different ways to categorize and tag files across projects though; it's a bit of a pain. [15:07:28] Gotcha. Cool. FWIW, I wouldn't mind seeing some simple metrics in the meantime. You can always test your presentation on us too :) [15:07:41] This is why we should have a Fair Use Commons. [15:08:17] halfak: Yeah, I war hoping to do that next week with bits of my tow presentations (one on those metrics, and the other one on the history analysis) [15:08:28] It has something for everyone. For those who are more permissive toward fair use, it makes it easier to use across different projects. For those who are less permissive, it offers ~*centralized control*~. It also makes metadata easier to manager, which I think we all like. [15:08:36] harej: No, this is why we should have structured data for files. [15:08:54] Are these mutually exclusive options? [15:09:01] No. [15:09:14] But I want structured data first :p [15:09:15] YuviPanda, I'm looking at the wikilabels fabfile. In order to create the schema in the DB, I'd like to run a 'wikilabels' utility. That means I need to install 'wikilabels'. Do you think it makes sense to follow the same process as the web machines? [15:10:40] halfak: is the 'wikilabels' utility part of the wikilabels package? [15:10:42] python package [15:10:52] yes [15:11:11] halfak: hmm, so maybe split that into setup_init setup_web setup_db? [15:11:18] +1 [15:11:24] That was my thought too. [15:11:27] Thanks. [15:32:11] YuviPanda, any thoughts on getting the OAuth credentials moved onto the wikilabels servers? [15:32:18] Right now, the creds live on my laptop. [15:32:19] ah, yes. [15:32:28] I think that's where they'll end up living [15:32:30] And get copied manually when necessary [15:32:33] let me find the one I use for quarry [15:32:33] OK [15:32:52] halfak: https://github.com/wikimedia/analytics-quarry-web/blob/master/fabfile.py#L36 [15:32:53] I figured we could have a fab "copy_oauth_creds" command or something [15:33:02] halfak: basically I do config merging - config is brought in by multiple files... [15:33:12] Oh... Hmmm [15:33:26] halfak: it's easy in flask! [15:33:31] flask.config automatically supports this by default [15:34:10] Hmm... Not using flask.config. I didn't know they had a standard mechanism for that. [15:34:48] Oh... looks like put() is really the solution. [15:35:12] I have a reference from the main config for where to find the oauth config. [15:35:30] So, I can just make sure that a put() gets called to move the config file where it needs to be. [17:42:22] does anyone know which table I should look for user ULS settings in? [17:52:32] leila: ask in #mediawiki-i18n ? Or ask a language developer directly, perhaps (Niklas, Amir, etc.). They certainly know. [17:52:45] thanks, guillom. I will. [17:52:48] I can't find it at first glance. [17:52:56] yeah. [17:53:48] I actually wondered if there was any DB setting at all. [17:54:07] I think ULS can work as an independent JS script, so maybe it's all JS and cookies.