[07:54:53] Catchy claim "low visibility of individual differences in online groups may allow them to harvest more of the benefits of diversity while bearing less of the cost" http://wikipapers.referata.com/wiki/The_effects_of_diversity_on_group_productivity_and_member_withdrawal_in_online_volunteer_groups [13:18:12] 10Quarry: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982#1325563 (10Halfak) 3NEW [13:20:41] 10Quarry: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982#1325577 (10Halfak) [13:59:06] o/ YuviPanda [13:59:10] hi halfak [13:59:38] I'm looking to set up an instance of Quarry for testing. It will be hitting another DB. Do you have a nice quick start guide for me? [13:59:48] * halfak crosses fingers [14:00:03] I'm trying to get people at the workshop I am to consider it as a potential dataset front-end. [14:00:18] workshop I am attending [14:00:20] * [14:02:17] halfak: ah. I don't have a quickstart, however :| [14:02:21] halfak: when is your workshop? [14:02:27] Now :) [14:02:41] ah, ow [14:02:59] I was going to do that but never quite got around to it. but general steps *should* work... [14:03:12] How much trouble would I have in starting it up on a new server and pointing it to a postgres DB? [14:03:13] it isn't tied to wikimedia servers [14:03:30] I don't need a LB or anything like that. [14:04:15] yeah [14:04:21] you just have to setup celery [14:07:44] Would celery's quickstart be sufficient? [14:08:57] halfak: yup [14:09:02] halfak: we aren't doing anything special [14:09:09] halfak: but celery does need a queue backend so you'd need redis as well [14:09:14] Cool. [14:09:17] No problem. [14:09:35] * halfak has done redis-based celery quickstart a few times [14:09:37] :) [14:10:08] halfak: +1 and document it :D [14:10:29] Hokay [14:11:17] halfak: in the long run, I want to have a docker image or something so people can just run one command and it'll set it all up for htem [14:11:48] Would be wonderful. [14:12:01] * halfak showed people the wonders on quarry and they seem to have bought in. [14:12:18] Also, I started getting a bunch of verbal feature requests and am trying to direct them to phab. [14:13:42] halfak: I saw a few! [14:13:46] wooo we found a milimetric :) [14:14:04] :) hey yall [14:14:06] what's up [14:14:57] welcome back :) [14:16:51] I was gone? [14:19:41] mabe [14:19:49] halfak: I need to spend a week on quarry at some point [14:20:03] my 'spare' cycles have been soaked up by ORES / Mesos / Marathon / girlfriend atm [14:20:19] YuviPanda: did you know about pymysql and how even SQLAlchemy supports it and how you can migrate to python 3 now? [14:20:39] yes :) I actually use pymyqsl in quarry [14:20:49] because it supports multiple resultsets the best [14:20:56] cool, I just found out about it, it's great [14:21:05] or was I using it and then moved away because it didn't support it? [14:21:06] * YuviPanda is unsure [14:21:12] YuviPanda, +1 [14:21:14] wow, it's been many months since Quarry was first built [14:21:33] Would like to coordinate a slow week where we can clean some things up together :) [14:21:45] o/ milimetric [14:21:45] halfak: maybe I should take a sabbatical from the ops team and come work with research (not analytics :P) for a quarter some point in the future :) [14:21:55] YuviPanda, ooooOOOOooo [14:21:55] :) [14:21:57] maybe not a quarter. a month maybe [14:22:03] still, woudl be fun [14:22:08] That'd be nice. :) [14:22:13] and I bet you / others will have plenty of ideas of what I can do [14:27:03] YuviPanda, any protips for running a celery worker automatically [14:27:05] ? [14:27:29] let me look at the command I use [14:27:49] I just found http://celery.readthedocs.org/en/latest/tutorials/daemonizing.html [14:29:23] i am an agent for change! muahahaha [14:29:26] but seriously, don't use python 2 [14:29:33] if anything, you should be using python 4 [14:29:42] halfak: looking [14:31:05] I'm looking at https://github.com/celery/celery/blob/3.1/extra/generic-init.d/celeryd [14:31:15] It seems like I should be able to just drop that in place -- I think. [14:32:07] halfak: celeryd --broker redis://localhost/0 --app quarry.web.worker [14:32:14] halfak: that should work [14:32:20] and give you one worker unit, I think [14:32:32] YuviPanda, we'd need to run that on restart, right? [14:32:40] halfak: this is just for testing, right? [14:32:45] Yeah... mostly [14:32:51] :) [14:32:56] :D [14:33:02] I think people might get grumpy if it stops working later. [14:33:06] halfak: yeah, if you're running it for production, the second link is what you want. [14:33:14] it's slightly complicated and what not [14:33:16] But I can point them to wmflabs quarry if they want a resilient demo. [14:33:24] yes [14:33:29] kk [14:33:32] it's puppetized there, and I'm dockerizing it now [14:33:59] How do I clone quarry from gerrit? Or should I just clone the github? [14:34:19] Sorry for the bother. Tell me "go read" at any point. [14:36:04] * halfak opts to not ping :) [14:36:15] Looks like github has recent commits. [14:36:18] I'll do that [14:39:11] halfak: yes, it's from gerrit but replicated to github [14:39:17] halfak: and do ping - I don't mind at all :) [14:39:20] * halfak looks for setuo.py [14:39:22] I jus tdon't read non-ping [14:39:33] there isn't one >_> [14:39:39] there's a requirements.txt tho [14:39:48] Do I need to uwsgi & stuff to run the server?? [14:40:22] halfak: in a 'production' level? yeah. if not you can just use python quarry.wsgi once I merge https://gerrit.wikimedia.org/r/#/c/215023/ [14:40:35] Cool. [14:40:38] (Merged) [14:40:47] :D [14:45:06] YuviPanda, (1) "python quarry.wsgi & " (2) " halfak: you probably want ot change some settings in quarry/config.yaml [14:46:02] halfak: specifially - output_path_template and the db settings [14:46:26] looks like that's not in the repo? [14:46:40] Oh! Wait I see it. [14:47:17] Should this support psql out of the box? [14:47:25] halfak: no, sadly. it just does mysql by default [14:49:48] Turns out that is no problem [14:51:11] * halfak makes some template data project folders [14:59:52] Time to go. [14:59:54] o/ [15:00:07] (got quarry started BTW -- didn't get a request to it from the outside yet) [15:00:22] halfak: \o/ sweet [15:00:29] halfak: write it up when you get the time :) [15:08:20] _o/ [15:08:50] guillom: o/ [15:09:00] guillom: have you seen harej's work on wikiprojects? seems quite great :D [15:09:51] YuviPanda: I've heard bits and pieces. I'm waiting to see the cool outcome :) [15:10:08] guillom: https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Directory/Culture_WikiProjects#Animation_WikiProjects [15:10:20] such latency on my local internet connection. boo [15:10:50] The only authoritative directory is /All; the categorized subcategories rely on shitty metadata. [15:10:52] Ooooh, nice! [17:58:54] ggellerman_____: confirmed I have to cancel/push our syncup with kevin [18:06:02] Thanks for letting me know...good luck with travel plans [19:04:19] 10Quarry: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982#1326793 (10Capt_Swing) What if the query is changed between executions? Do we save the version the query that produced each result set? [19:10:45] 10Quarry: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982#1326827 (10yuvipanda) Yes we already do have revisions - they are saved each time the query is executed :) We do need to expose them.