[00:06:27] (PS1) Milimetric: Add an option to schedule reports for all cohorts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/150440 [00:46:27] (PS1) Yuvipanda: Add default_config.py with some sane default values [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150453 [00:52:26] (CR) Legoktm: [C: 2] Add default_config.py with some sane default values [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150453 (owner: Yuvipanda) [00:52:31] (Merged) jenkins-bot: Add default_config.py with some sane default values [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150453 (owner: Yuvipanda) [01:07:59] Analytics / Wikimetrics: session management - https://bugzilla.wikimedia.org/68833 (Dan Andreescu) NEW p:Unprio s:normal a:None There MUST be a session not being closed somewhere, as while doing a lot of testing on wikimetrics, I encountered: OperationalError: (OperationalError) (1203, "Us... [01:51:36] (PS1) Yuvipanda: Add wsgi file for uwsgi deployment convenience [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150467 [01:52:24] (CR) Yuvipanda: [C: 2] Add wsgi file for uwsgi deployment convenience [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150467 (owner: Yuvipanda) [01:52:30] (Merged) jenkins-bot: Add wsgi file for uwsgi deployment convenience [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150467 (owner: Yuvipanda) [01:54:24] Analytics / Wikimetrics: Wikimetrics can't run a lot of recurrent reports at the same time - https://bugzilla.wikimedia.org/68840 (Dan Andreescu) NEW p:Unprio s:normal a:None When adding reports for all the wiki cohorts and manually kicking off the scheduler, not all reports could successfu... [02:51:21] Analytics / Wikimetrics: Wikimetrics can't run a lot of recurrent reports at the same time - https://bugzilla.wikimedia.org/68840#c1 (Dan Andreescu) I think I found the problem. Celery chains are not documented very well. Apparently, if one of their children raises an error, the chain can stop: http... [02:51:23] (PS1) Milimetric: Fix report chain stopping [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/150475 (https://bugzilla.wikimedia.org/68840) [03:10:19] (CR) Milimetric: [C: -1] "no tests and not really confirmed working. This change needs a lot of work. One problem that happened already is that it causes the writ" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/150475 (https://bugzilla.wikimedia.org/68840) (owner: Milimetric) [05:41:53] Analytics / Wikimetrics: Bring WIKIMETRICS_POOL_SIZE to vagrant's wikimetrics setup - https://bugzilla.wikimedia.org/68825 (christian) [10:36:52] (PS1) Yuvipanda: Add missing __init__.py file to quarry [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150513 [10:39:28] (PS1) Yuvipanda: Create the quarry database if it doesn't exist [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150514 [11:01:58] (PS1) Yuvipanda: Use redis for sessions [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150523 [11:04:46] (PS2) Yuvipanda: Use redis for sessions [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150523 [12:56:47] (CR) Yuvipanda: [C: 2] Add missing __init__.py file to quarry [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150513 (owner: Yuvipanda) [12:56:53] (Merged) jenkins-bot: Add missing __init__.py file to quarry [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150513 (owner: Yuvipanda) [12:57:02] (CR) Yuvipanda: [C: 2] Create the quarry database if it doesn't exist [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150514 (owner: Yuvipanda) [12:57:07] (Merged) jenkins-bot: Create the quarry database if it doesn't exist [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150514 (owner: Yuvipanda) [12:57:15] (CR) Yuvipanda: [C: 2] Use redis for sessions [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150523 (owner: Yuvipanda) [12:57:20] (Merged) jenkins-bot: Use redis for sessions [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150523 (owner: Yuvipanda) [12:57:45] (PS1) Yuvipanda: Fix License notice to be consistently CC0 [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150540 [13:04:10] (PS1) Yuvipanda: Minor fixes to landing page [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150542 [13:28:24] Analytics / Tech community metrics: Wrong data at "Update time for pending reviews waiting for reviewer in days" - https://bugzilla.wikimedia.org/68436#c5 (Quim Gil) Also, DataValues is now the #1 repo, but I can't find any repo by this name, neither open reviews in the three related projects: https:/... [13:31:41] milimetric: I puppetized 'em all :) [13:31:49] running on two machines from the start now [13:32:00] nice [13:32:28] side note: puppet will restart celery, so make sure you can handle that [13:32:34] well, depends on how you have it set up [13:32:53] but for us, it restarts it and currently all the workers get lost and confused and it causes a lot of headache [13:33:46] milimetric: ah, no, it doesn't restart it for me. [13:33:50] milimetric: I'm doing deployments via fabric [13:33:55] milimetric: so I get control over when those happen [13:34:18] milimetric: also I'll probably turn on acks_late for celery as well, so restarting celery doesn't restart everything [13:34:24] err, *does* restart all the jobs [13:35:05] milimetric: I was going to do the deployment with puppet as well, but felt too awkward and inelegant and I couldn't automate a way of storing private config [13:35:31] milimetric: and I talked to springl.e, he said 10m timeouts are good enough :) [13:36:03] ah ok, fabric deploy seems like a better way to go [13:36:17] i have no love for puppet deploy after all the trouble [13:39:18] milimetric: heh, yeah. puppet for deployment just seems like a bad idea for everyhing that's not super trivial [13:39:46] milimetric: now I just need to implement fork, CSV download, and am good to go :) [13:41:16] sweet man, seems like great progress that will be super useful at wikimania [13:41:26] note to self: hack before the hackathon :) [14:12:57] milimetric: :D [14:13:54] heya qchris_meeting, do you think we should adapt these queries and stick them in refienry repo? [14:13:57] just to have them there? [14:13:57] https://gist.github.com/ottomata/7048012 [14:13:59] they are useful [14:38:09] ottomata: Sure! If they are useful, we should totally add them to refinery. [15:57:06] Analytics / Wikimetrics: session management - https://bugzilla.wikimedia.org/68833#c1 (Dan Andreescu) looking through the labsdb databases, here's a list of the wikis to which connections were zombied while running RollingActiveEditor: svwiki ruwiki lgwiki nostalgiawiki lijwiki vowiktionary swwiki glw... [16:17:43] (PS1) Ottomata: Add select_missing_sequence_runs.hql [analytics/refinery] - https://gerrit.wikimedia.org/r/150569 [16:59:55] qchris you have so many hadoop workers in labs! :p [17:00:02] i think we may be at our quota [17:00:09] oh he's not here, hm [19:36:18] milimetric: hey! sorry to have gotten wikimetrics into that [19:37:27] oh not at all YuviPanda|Bus, I think it's good to have these talks out [19:37:40] and I only jumped in originally to try to help with your use case [19:37:49] but I'm happy with whatever ops decides, as long as it's consistent [19:37:52] milimetric: yeah :) I guess I'll end up learing to build packages [19:38:11] milimetric: which is probably not too bad. last time I got a lecture from ops, I started learning puppet :) [19:42:08] :) debian packaging is cool, i honestly wish the whole world just revolved around debian [19:42:15] like, that npm didn't come up with their own thing [19:42:19] but alas [19:43:26] milimetric: heh [19:43:37] milimetric: yeah, maybe. it's just a bit too high standards, I suppose [21:20:06] Analytics / Wikimetrics: Story:b WikimetricsUser runs 'Rolling New Active Editors' report - https://bugzilla.wikimedia.org/67459 (Kevin Leduc) p:Highes>Normal [21:20:51] Analytics / Wikimetrics: Story:c WikimetricsUser runs 'Rolling Surviving New Active Editors' report - https://bugzilla.wikimedia.org/67460 (Kevin Leduc) p:Highes>Normal [21:40:36] Analytics / Wikimetrics: Spike: Mediawiki Dashboard Storage - https://bugzilla.wikimedia.org/68448 (Kevin Leduc) p:Unprio>High [21:44:51] Analytics / Wikimetrics: Story: WikimetricsUser generates report with Target Site breakdown - https://bugzilla.wikimedia.org/68475 (Kevin Leduc) p:Unprio>Low [21:49:35] Analytics / Wikimetrics: Story: WikimetricsUser generates report with Target Site breakdown - https://bugzilla.wikimedia.org/68475#c2 (Kevin Leduc) The research team believes there is a way to generate these metrics without using EL data. I'll be looking to them to generate the initial SQL the same wa... [21:51:20] Analytics / Visualization: Story: EEVSUser selects Target Site breakdown - https://bugzilla.wikimedia.org/68473#c1 (Kevin Leduc) p:Unprio>Low See Pau's design... but instead of a "Devices" breakdown (bottom left column), it's a target site breakdown. http://pauginer.github.io/prototypes/analytics... [21:55:21] Analytics / Wikimetrics: Deploy "rolling active editors" metric to production. - https://bugzilla.wikimedia.org/68447 (Kevin Leduc) p:Unprio>Normal [21:57:21] Analytics / Wikimetrics: Story: WikimetricsUser shares a cohort - https://bugzilla.wikimedia.org/67461 (Kevin Leduc) p:Normal>Lowest [21:57:35] Analytics / Wikimetrics: Epic: WikimetricsUser shares a cohort - https://bugzilla.wikimedia.org/67461 (Kevin Leduc) [22:02:35] Analytics / Wikimetrics: Story: WikimetricsUser runs report against all wikis - https://bugzilla.wikimedia.org/68477 (Kevin Leduc) p:Unprio>Low [22:16:51] Analytics / Visualization: Story: EEVSUser selects ALL wikis - https://bugzilla.wikimedia.org/68478 (Kevin Leduc) p:Unprio>Low [22:18:06] Analytics / Wikimetrics: Cohort listing going against the database instead of using CohortService - https://bugzilla.wikimedia.org/66126 (Kevin Leduc) p:Normal>Lowest [22:20:51] Analytics / Refinery: Story: AnalyticsEng has UDF in Hadoop for UA parsing - https://bugzilla.wikimedia.org/67803 (Kevin Leduc) p:Highes>Low [22:23:06] Analytics / Visualization: Story: AnalyticsEng has website for EEVS - https://bugzilla.wikimedia.org/68351 (Kevin Leduc) p:Unprio>Highes [23:48:08] Is anyone from analytics around? [23:48:18] couldn't find anyone in the office [23:50:11] we're here dude [23:53:21] kaldari, yo dawg [23:53:24] I'm at my desk [23:53:29] howdy! [23:53:38] come say hi, or something. [23:53:47] if you still need analytics [23:59:21] Ironholds: nevermind, Maryana solved the mystery!