[05:54:29] hello there, i had a query regarding unmatched search results on wikipedia [05:56:26] is any staistical data available on searches which result in non-exact matches? [06:42:10] shubnam: I do not think we have data to that extent but you should ask on the analytics e-mail list [06:42:34] shubham: I do not think we have data to that extent but you should ask on the analytics e-mail list [06:43:01] ^shubham [10:30:54] Analytics / General/Unknown: Kafka broker analytics1021 not receiving messages since 2014-08-06 ~1:44 - https://bugzilla.wikimedia.org/69244#c6 (Toby Negrin) Thanks Gage -- I'm really wondering whether there's a specific problem with this host, like a hardware issue. -Toby [11:07:27] (PS1) Nuria: [WIP] Showing usage of celery signals Code can be used to manage db session scope [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/152888 [11:07:34] (CR) jenkins-bot: [V: -1] [WIP] Showing usage of celery signals Code can be used to manage db session scope [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/152888 (owner: Nuria) [11:19:57] (PS2) Nuria: [WIP] Showing usage of celery signals Code can be used to manage db session scope [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/152888 (https://bugzilla.wikimedia.org/68833) [11:20:04] (CR) jenkins-bot: [V: -1] [WIP] Showing usage of celery signals Code can be used to manage db session scope [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/152888 (https://bugzilla.wikimedia.org/68833) (owner: Nuria) [11:20:56] Analytics / Wikimetrics: session management - https://bugzilla.wikimedia.org/68833#c7 (nuria) see sample patch: https://gerrit.wikimedia.org/r/#/c/152888/ [12:20:25] Analytics / EventLogging: Cleaning up of some (?) EventLogging schemata for Growth - https://bugzilla.wikimedia.org/68931#c5 (christian) As discussed in private emails between Steven, Aaron and me, the request is only for the following schemas: SignupExpAccountCreationComplete SignupExpAccountCrea... [12:20:54] Analytics / EventLogging: Cleaning up of some (?) EventLogging schemata for Growth - https://bugzilla.wikimedia.org/68931#c6 (christian) The tables to be purged from the log database are SignupExpAccountCreationComplete_8539421 SignupExpAccountCreationImpression_8539445 SignupExpCTAButtonClick_8... [12:56:59] (Abandoned) Yuvipanda: Minor fixes to landing page [analytics/quarry/web] - https://gerrit.wikimedia.org/r/150542 (owner: Yuvipanda) [13:35:29] (PS4) Milimetric: Put focus on last connection in connection pool test [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/149303 (owner: QChris) [13:35:35] (CR) Milimetric: [C: 2] Put focus on last connection in connection pool test [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/149303 (owner: QChris) [13:35:59] (Merged) jenkins-bot: Put focus on last connection in connection pool test [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/149303 (owner: QChris) [13:48:29] \o/ milimetric [13:48:37] :) [13:48:40] hi nuria! [13:54:37] so milimetric, dario must have given his talk alredy right? [13:54:53] yea, very well could be [14:48:09] Analytics / Wikimetrics: Backing up wikimetrics data fails if data is written while we back it up - https://bugzilla.wikimedia.org/68731#c2 (Kevin Leduc) p:Unprio>Highes s:normal>enhanc collaboratively tasked on etherpad: http://etherpad.wikimedia.org/p/analytics-68731 [14:56:38] Analytics / Wikimetrics: replication lag may affect recurrent reports - https://bugzilla.wikimedia.org/68507 (Kevin Leduc) p:Highes>Normal [15:02:53] Analytics / Wikimetrics: session management - https://bugzilla.wikimedia.org/68833 (Kevin Leduc) [15:14:26] Analytics / Wikimetrics: Fix admin script so it does not schedule reports for 'nonexisting' databases - https://bugzilla.wikimedia.org/69297 (nuria) NEW p:Unprio s:normal a:None Fix admin script so it does not schedule recurrent reports for 'nonexisting' databases. [15:22:54] Analytics / Wikimetrics: Wikimetrics can't run a lot of recurrent reports at the same time - https://bugzilla.wikimedia.org/68840 (Kevin Leduc) [15:34:25] Analytics / Wikimetrics: Recurrent runs for project cohorts should not block reports scheduled by users - https://bugzilla.wikimedia.org/69299 (nuria) NEW p:Unprio s:normal a:None Recurrent runs for project cohorts should not block reports scheduled by users.Ideally "user" scheduled tasks s... [15:37:10] Analytics / Wikimetrics: Story: AnalyticsEng has Execution Pipeline to optimize/create resusable code for generating metrics - https://bugzilla.wikimedia.org/69300 (Kevin Leduc) NEW p:Unprio s:enhanc a:None * (8 points) prep & design meetings, decide how to implement. factors: ** testing d... [15:38:08] Analytics / Wikimetrics: Story: AnalyticsEng has Execution Pipeline to optimize/create resusable code for generating metrics - https://bugzilla.wikimedia.org/69300 (Kevin Leduc) p:Unprio>Lowest [15:39:18] milimetric: around?? [15:52:26] Analytics / Wikimetrics: Need to create a permanent and vetted version the "editor_day" table - https://bugzilla.wikimedia.org/69145 (Kevin Leduc) [15:57:12] yuvipanda: we are in a sprint tasking meeting [15:57:15] need help? [15:57:31] nuria: ah, cool. Nothing urgent :) [15:57:46] is everyone loving your quarry tool? [15:58:07] nuria: indeed :D [15:58:36] nuria: someone made a logo https://commons.wikimedia.org/wiki/File:Quarry-logo.svg [15:58:47] jajaj [15:58:50] it's great! [15:59:25] nuria: :D [16:12:07] (PS4) Phuedx: WIP Migrate models to SQLAlchemy [analytics/quarry/web] - https://gerrit.wikimedia.org/r/152751 [16:12:13] (CR) jenkins-bot: [V: -1] WIP Migrate models to SQLAlchemy [analytics/quarry/web] - https://gerrit.wikimedia.org/r/152751 (owner: Phuedx) [16:18:08] Analytics / Wikimetrics: Story: AnalyticsEng uses TimeSeries support to backfill data - https://bugzilla.wikimedia.org/69253 (Kevin Leduc) [16:19:59] Analytics / Wikimetrics: Story: AnalyticsEng has editor_day table in labsdb - https://bugzilla.wikimedia.org/69254#c2 (Kevin Leduc) NEW>RESO/DUP *** This bug has been marked as a duplicate of bug 69145 *** [16:19:59] Analytics / Wikimetrics: Need to create a permanent and vetted version the "editor_day" table - https://bugzilla.wikimedia.org/69145#c2 (Kevin Leduc) *** Bug 69254 has been marked as a duplicate of this bug. *** [16:20:29] Analytics / Wikimetrics: Story: AnalyticsEng has editor_day table in labsdb - https://bugzilla.wikimedia.org/69145 (Kevin Leduc) [16:27:40] Analytics / Wikimetrics: Story: AnalyticsEng has static file with list of projects and metrics - https://bugzilla.wikimedia.org/68822 (Kevin Leduc) [16:58:08] Analytics / Wikimetrics: Story: EEVS user does not see reports for projects without databases - https://bugzilla.wikimedia.org/69297 (Kevin Leduc) [16:59:08] Analytics / Wikimetrics: Story: EEVS user does not see reports for projects without databases - https://bugzilla.wikimedia.org/69297#c1 (Kevin Leduc) p:Unprio>Normal s:normal>enhanc collaborative tasking on etherpad: http://etherpad.wikimedia.org/p/analytics-69297 [17:19:21] ok yuvipanda, around now :) [17:21:07] hey [17:21:41] oops, wrong channel [19:27:34] milimetric, do you know who DSC is on this graph? https://github.com/wikimedia/limn/network [19:27:41] is it even worth merging? [19:28:10] the code seemed to have significantly drifted apart [19:28:38] dsc == david [19:28:49] not worth merging, but maybe using for inspiration [19:32:14] yurikR: ^ [19:33:00] milimetric, thx. What about limm.js - is there a place i could simply copy that file into the wikimedia/extensions/limn ? [19:33:18] limn.js? [19:33:37] that's what readme said - i don't need node.js to run limn [19:33:46] simply include limn.js and done [19:33:58] heh, it says that? :) not sure about that... [19:34:26] oh i see [19:34:35] that's more aspirational than anything yurikR, sorry about that [19:35:11] are you saying i must have node.js to run limn? [19:36:46] yurikR: one sec [20:00:03] ah, sorry yurikR [20:00:16] I can give you my undivided attention now [20:00:32] limn does not "need" anything, in the strict sense of "need" [20:00:39] it can run in just the browser [20:01:00] but it looks for /datasources/.json|yaml [20:01:21] milimetric, right, so if i wanted to add limn in its entirety to the limn extension, is there a one compiled (but not minified) file? [20:01:33] and then /data/datafiles// [20:02:00] i wouldn't want the extension and limn itself mixed up [20:02:13] you can build a compiled file, but again, you would need to make it get its metadata from somewhere other than the limn server [20:02:26] you can get the compiled file like this: [20:02:27] coke build [20:02:29] coke bundle [20:02:43] (here's the file for the reportcard: http://reportcard.wmflabs.org/js/limn.no-deps.js) [20:03:03] (and this is all the dependencies: http://reportcard.wmflabs.org/vendor/vendor-bundle.js) [20:04:13] so doing that and putting it in the extension would not work, you'd have to change how it renders the graphs, then how the graphs fetch the datasources, and how the datasources fetch the data [20:04:27] in short, it might be a lot easier to not use limn, and just use rickshaw or something [20:04:55] whereas limn was *meant* to do this kind of thing, it was aborted in early 2013 and never really given a chance I think [20:05:10] yurikR: ^ [20:05:31] that implies i have a mac which i don't ;) [20:06:20] sigh. I want the niceties of limn - graphs look much much better [20:06:58] yurikR: I don't have a mac - coke is "make" for coco and you can get it with "sudo npm install -g coco" [20:07:37] thx. Might play with it soon. rickshaw looks very promising too ) [20:07:59] i agree limn graphs look much better, and I promise I'll port that look, but I personally think limn, its 500K of dependencies, and its over-complicated architecture are not worth the style it brings [20:08:01] style is easy [20:08:21] rickshaw is great, gets you all the features you'd ever want from limn [20:08:42] theoretically, so does vega, but that's a bit harder to style as it has a more rigid grammar (good and bad) [20:12:03] milimetric, from what i see in rickshaw, it does not support data-driven drawing - all samples rely on javascript setting up the view [20:12:28] which might be good for fully-trusted content, but not in the wiki setting [20:12:54] hm, weird, lemme take a closer look [20:14:29] right yurikR, it's lower level than limn, you'd have to build up from the pieces it gives [20:14:40] sigh :) [20:14:48] but I mean, you see the difficulties with limn [20:15:15] not to mention that firefox's most recent updates broke some compatibilities, updating to the latest versions of d3 seems to break it, etc. [20:15:36] limn has literally not received any actual dev for the last 16 months [20:16:25] so which lib do you think we will use moving forward - vega? And should we push it to main WP as a way to do arbitrary graphs/maps? [20:16:58] yurikR: yes, I think vega solves everything limn tried to solve in the visualization space [20:17:06] we're working on dashiki to solve what limn tried to do in the dashboarding space [20:17:19] and those two combined should meet all the needs I've heard over the past two years [20:17:39] vega is *totally* capable of doing arbitrary visualizations on-wiki [20:18:01] the only thing it's really missing is a bit of interactivity built on top of it [20:18:09] zooming, hovering, etc. [20:18:23] milimetric, so do you think the limn extension should eventually be pushed to all WP for articles to use? [20:18:27] and their latest version, from 1.4 on I think, makes that possible, I just have to write some simple adapters [20:18:30] (vega based) [20:18:42] yurikR: yes, definitely [20:18:51] It's just not high priority for analytics right now [20:19:04] ok, in that case i will start developing it for zero and push it into production [20:19:10] because we're focusing on getting vital signs broken down by project across all our projects [20:19:28] once its on zerowiki, we can easily move it to other prod wikies [20:20:01] that sounds good yurikR, and I think style-wise a lot can be done. Interactivity-wise, we were going to reach out to the vega folks [20:20:13] but do you need that stuff right now? Or is a non-interactive graph ok to start with? [20:20:38] style-wise, just play with the marks definitions and the options you get there, it's quite flexible [20:21:12] I'll play with zoom and hover this weekend and see if I can whip something up [20:21:13] milimetric, well, currently our partners look at the nice limn graphs. It would kinda suck to say "ok, we can no longer provide it, but here are some not-as-good,unstyled graphs" [20:21:33] :/ yeah, def. [20:21:51] but i do want to do it because that makes everything much easier to move forward with [20:22:12] without this step, we are stuck with the current graphs and can't do much more [20:22:25] hence - i'm ok to make it temporarily not as pretty [20:22:38] as long as we know we will improve them soonish :) [20:22:51] (btw, i'm horrible with styling, hence my hesitance) [20:22:55] ok, I'll try to play with this stuff more urgently then, on weekends and stuff [20:22:59] pretty vega - coming up [20:23:05] you rock :) [20:23:19] don't push yourself too hard, weekends are for resting ;) [21:34:10] (PS5) Phuedx: Migrate models to SQLAlchemy [analytics/quarry/web] - https://gerrit.wikimedia.org/r/152751 [21:34:15] (CR) jenkins-bot: [V: -1] Migrate models to SQLAlchemy [analytics/quarry/web] - https://gerrit.wikimedia.org/r/152751 (owner: Phuedx) [21:41:07] (PS6) Phuedx: WIP Migrate models to SQLAlchemy [analytics/quarry/web] - https://gerrit.wikimedia.org/r/152751 [21:44:21] yuvipanda: so the worker needs to be modified and then we're done [22:02:33] phuedx / yuvipanda: congrats on quarry, I heard it was super well received [22:03:02] milimetric: it was all yuvipanda, i'm just helping him out :) [22:03:06] I look forward on helping / improving the schema of data people need to query (but of course I'm tied up until after this quarter) [22:03:20] (boo to being tied up) [22:04:00] I honestly wish I could clone myself or become 10x faster somehow. Sadly, my 20s are behind me :) [22:07:18] hehe [22:07:34] i know exactly how you feel [22:23:54] (PS7) Phuedx: WIP Migrate models to SQLAlchemy [analytics/quarry/web] - https://gerrit.wikimedia.org/r/152751 [22:24:27] worker done [22:24:40] a little tidy up tomo and i'll take the wip off [22:24:44] g'night folks [23:36:39] Analytics / Wikimetrics: Backing up wikimetrics data fails if data is written while we back it up - https://bugzilla.wikimedia.org/68731 (Kevin Leduc) [23:37:23] Analytics / Wikimetrics: session management - https://bugzilla.wikimedia.org/68833 (Kevin Leduc) [23:37:38] Analytics / Wikimetrics: Wikimetrics can't run a lot of recurrent reports at the same time - https://bugzilla.wikimedia.org/68840 (Kevin Leduc) [23:38:08] Analytics / Wikimetrics: replication lag may affect recurrent reports - https://bugzilla.wikimedia.org/68507 (Kevin Leduc) [23:39:23] Analytics / Visualization: Story: EEVSUser loads static site in accordance to Pau's design - https://bugzilla.wikimedia.org/67806 (Kevin Leduc) [23:51:08] Analytics / EventLogging: Cleaning up of some (?) EventLogging schemata for Growth - https://bugzilla.wikimedia.org/68931 (Kevin Leduc) [23:51:23] Analytics / Wikimetrics: Rolling Active Editor is slow - https://bugzilla.wikimedia.org/68596 (Kevin Leduc) [23:56:23] Analytics / Visualization: Story: EEVSUser loads static site in accordance to Pau's design - https://bugzilla.wikimedia.org/67806 (Kevin Leduc)