[00:21:31] (PS1) Nuria: Adding index on cohort_wiki_user [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) [00:21:39] (CR) jenkins-bot: [V: -1] Adding index on cohort_wiki_user [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) (owner: Nuria) [00:24:42] (PS2) Nuria: Adding index on cohort_wiki_user [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167134 (https://bugzilla.wikimedia.org/71255) [03:39:54] Analytics / EventLogging: VE instrumentation is not showing up in databases - https://bugzilla.wikimedia.org/72173 (Toby Negrin) NEW p:Unprio s:normal a:None VE has a variety of schemas[1] that are not correctly feeding into the EL databases. They need assistance in figuring out what's happ... [09:59:05] (PS1) QChris: Add overview diagram for Oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/167184 [10:02:10] (PS1) QChris: Add overview diagram for Oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/167185 (https://bugzilla.wikimedia.org/71994) [10:28:02] (PS2) QChris: Add overview diagram for Oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/167185 (https://bugzilla.wikimedia.org/71994) [10:28:50] (Abandoned) QChris: Add overview diagram for Oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/167184 (owner: QChris) [10:35:38] Analytics / Refinery: No new Pagecounts-all-sites files since 2014-10-10 17:00 - https://bugzilla.wikimedia.org/71994#c6 (christian) (In reply to Andrew Otto from comment #4) > Was there a coordinator we had to > manually submit too? Yes, the Oozie job that fills webstats table is only a coordinator.... [14:56:16] kevinator: arg, I forgot - when looking at the edits/pages metrics, did we decide to change the default parameters? [14:56:22] to include all namespaces? [14:56:30] it doesn't say that in the etherpad, but I remember you agreeing to it [14:57:24] yes, I agreed to it [15:00:34] k, thx [15:06:25] (PS1) Milimetric: Default to all namespaces for edits and pages [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167214 [15:36:43] (CR) Nuria: "Doesn't this need corresponding UI changes so users can choose whether to look just at namepsace 0?" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167214 (owner: Milimetric) [15:39:49] nuria: the namespace field is a textbox that accepts a comma-separated list [15:40:02] when it's blank, it means "all" [15:40:12] if they want just namespace 0, they just type in "0" [15:40:19] or am I missing your point? [15:41:33] Milimetric: no you are right, i just forgot the box was displayed already [15:41:45] k, i'll comment on the change, just making sure [15:42:28] (CR) Milimetric: "The namespace field is displayed on the page as a text box, namespace 0 can be selected just as before, by typing it into the box. But we" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167214 (owner: Milimetric) [15:42:57] (PS2) Milimetric: Default to all namespaces for edits and pages [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167214 (https://bugzilla.wikimedia.org/72114) [15:44:39] nuria: I'm thinking let's let mforns review and merge this, then I'll do the deploy together with him (through staging first) [15:45:04] milimetric: ok, let me test it a bit [15:47:34] (CR) Nuria: [C: 1] Default to all namespaces for edits and pages [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167214 (https://bugzilla.wikimedia.org/72114) (owner: Milimetric) [16:04:53] Analytics / General/Unknown: kafkatee not consuming for some partitions - https://bugzilla.wikimedia.org/71056#c4 (Andrew Otto) Ok! Magnus says that he found the bug and fixed it. https://github.com/edenhill/librdkafka/commit/7c151bcac2230c957b40816f3fb333d729ee3dc7 I've started kafkatee back up a... [16:57:52] Analytics / EventLogging: VE instrumentation is not showing up in databases - https://bugzilla.wikimedia.org/72173#c1 (Dan Andreescu) No new events are coming in for schemas like VisualEditor%: zsub vanadium.eqiad.wmnet:8600 | grep '"schema": "VisualEditor' But at least two of the events used to vali... [17:32:04] oh J-Mo I wanted to ask an unrelated question [17:32:19] sure, shoot milimetrc [17:32:27] we're thinking of changing the default parameter for the Edits and Pages metrics [17:32:30] (pages created) [17:32:37] right now, they have namespaces = 0 [17:32:45] and we want to make it namespaces = blank (blank == all) [17:33:19] * J-Mo is looking at wikimetrics… [17:33:22] so the impact is that this will increase the numbers for any reports you run in the future, unless you just manually set it to namespace = 0 [17:34:05] the reason we want to do this is because the research team defined these defaults as the ones that make sense for these metrics [17:34:43] and with the current implementation it's weird to have two defaults - one for people using it from the UI and one for people scheduling recurrent Vital Signs reports [17:34:53] possible to get some grey hint text in the blank text field, explaining that 'all' is the default? [17:35:19] since I assume the namespace textbox will now be empty when you create a new report [17:35:44] did you see the placeholder text if you make the textbox empty J-Mo? [17:35:47] is that enough? [17:35:58] ah, bene! [17:35:59] (that's what would show up after the proposed change) [17:36:04] ok, cool [17:36:22] yes, that's perfect. I don't see a problem with changing the default. Will Kevin make the announcement to the Wikimetrics list? [17:36:26] we'll make it so and make sure to communicate it super clearly on the lists. I'll try to run some reports with some random cohorts to get a feel for what the difference would be [17:36:36] sounds good to me [17:37:15] Hey folks. I have a prof at the U of Michigan who wants to use WikiMetrics on a local MediaWiki installation. Is the system designed to work with any mediawiki database? [17:44:50] halfak: yes [17:45:01] with a somewhat small snag [17:45:20] wikimetrics gets its "list of wikis" from meta (or the api soon) [17:45:50] Hmm... Could it get its list of wikis from another wiki or is meta hard-coded? [17:45:52] and in the case of external users, they would have to just change that one function. [17:46:04] OK. That seems do-able. [17:46:13] that code's pretty hard-coded, but it's a very simple python method [17:46:20] it could be hard-coded to just return "blah" [17:46:26] besides that, everything's configurable [17:46:53] so there's a get_mw_session that uses a templated url for how to get to the mediawiki database, and that template is configurable [17:46:54] How interested would you guys be in responding to bugs for a non-MediaWiki wiki? [17:47:18] um... moderately interested. We'd have to prioritize other things but if wikimetrics is useful to someone else, that's great! [17:47:30] That's what I'm hoping. [17:47:49] The guy running the class has bit of a grant to throw around if funding was an issue/opportunity. [17:47:55] and we accept patch requests of course - so for example if they want to make that method more generic, we'd merge that [17:48:06] Cool. [17:48:20] well, funding may be needed if he wants to stand it up and doesn't know python / what's involved [17:48:32] i'm not sure how well the puppet configuration that we use to deploy it will work for his situation [17:48:42] but it's possible to deploy manually of course [18:19:40] halfak: already in dev environment list of projects is overwriiten [18:19:58] sorry halfak, wikimetrics [18:20:04] gotcha [18:20:05] talk [18:20:12] cc milimetric [18:20:16] as in vagrant env [18:20:23] there is only one project 'wiki' [18:20:34] called 'wiki' [18:21:01] yep, that list is not configurable though, it still requires some tiny code changes [18:21:04] halfak: also list is overwritten (with a different set) for tests [18:21:23] milimetric: making it configurable should be pretty easy [18:21:32] agreed [18:31:30] mforns: i can't keep track of who knows what, but we decided to leave the CentralAuth cohorts the way they are [18:31:55] so business as usual for that - expand when we upload the cohort, and make it into a "fixed" cohor [18:31:56] ok [18:31:57] milimetric, mforns : i have sent notes of our meeting to internal-list [18:32:09] so we are all on teh same page, including kevinator [18:32:25] right - just making sure marcel knows as early as possible [18:34:25] milimetric, nuria: right [18:53:19] (CR) Mforns: [C: 1 V: 1] "For what I've seen, this makes sense!" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167214 (https://bugzilla.wikimedia.org/72114) (owner: Milimetric) [18:53:51] mforns: ok, so it looks like that change is ready to test in staging [18:54:03] your first priority is now the cohort thing, but let me know if you want to do a staging deploy / test [18:54:05] ok [18:54:19] how much time will it take? [18:55:31] hm, probably 10-20 minutes to test in staging, then i guess we'd have to wait until we announce it on the list to deploy to prod [18:55:56] but 30-60 minutes of deploying it to prod because that's when we have to clean up the db records and do all that [18:56:47] milimetric, mforns : also rememeber there are other things on master (like marcel's prior bug fixes) [18:56:53] so those two will get deployed too [18:57:15] yep [18:57:35] milimetric: so if you planed to deploy it today, I'm in [19:13:41] mforns: let's do it now! [19:13:43] to the batcave! [19:13:48] :] [19:28:26] milimetric: Nananananananananana BATMAAAAAAAN [20:15:51] (CR) Milimetric: "Marcel and I tested in staging. We found pretty different numbers with the new default but nothing broke. As a side comment, english wik" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/167214 (https://bugzilla.wikimedia.org/72114) (owner: Milimetric) [20:16:17] kevinator: see that last comment on https://gerrit.wikimedia.org/r/#/c/167214/, it pertains to performance of the edits metric in production [20:16:24] tl; dr; is that it might also be slow :( [20:21:07] got it that it might be slow. [20:21:40] one more reason for the data warehouse. [20:22:27] BTW are we missing a story in the backlog: once we have a data warehouse, we’ll need to refactor all the metrics to take advantage of it? [20:24:49] milimetric: ^^ [20:25:17] i think that was a task on one of the stories [20:25:25] one sec kevinator, looking [20:26:00] yep kevinator, part of the task: http://etherpad.wikimedia.org/p/analytics-69145 [20:26:26] it makes sense, because if we can't re-write the metrics to be much more performant, then the approach didn't work and that story's not done [20:29:17] The story above is just for RROAE. if it works, we’ll create a list of the remaining metrics to refactor. [20:29:56] ok, sounds fine to me [21:24:07] Analytics / EventLogging: VE instrumentation is not showing up in databases - https://bugzilla.wikimedia.org/72173#c2 (Roan Kattouw) Turns out EventLogging's mw.track subscriber is broken, and VisualEditor was using it for no good reason. I'll port VE to use EL directly, but there are other uses of mw.... [21:27:55] Analytics / EventLogging: ext.eventLogging.subscriber.js broken - https://bugzilla.wikimedia.org/72197 (nuria) NEW p:Unprio s:normal a:None File: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/modules/ext.eventLogging.subscriber.js The ext.eventLogging.subscrib... [21:47:09] Analytics / EventLogging: ext.eventLogging.subscriber.js broken - https://bugzilla.wikimedia.org/72197#c1 (Roan Kattouw) Specifically, what's broken is the way that it tries to derive the schema from the topic name. The things that are broken about it are: * It ends up trying to load schema..foo (doub... [22:05:23] Analytics / EventLogging: ext.eventLogging.subscriber.js broken - https://bugzilla.wikimedia.org/72197#c3 (Roan Kattouw) (In reply to Roan Kattouw from comment #1) > Specifically, what's broken is the way that it tries to derive the schema > from the topic name. The things that are broken about it are:... [22:40:15] hey, analytics engineers [22:40:25] can anyone recommend a way to hard-kill a unix process that's been Td but won't go away? [22:42:08] kill -9 [22:43:40] ta [22:44:19] holy crap how did I not know that. Thanks! [22:57:49] :) [22:57:55] later y'all, enjoy your weeknds [23:35:51] nuria: you around? [23:42:22] Analytics / EventLogging: VE instrumentation is not showing up in databases - https://bugzilla.wikimedia.org/72173#c3 (Roan Kattouw) (In reply to Roan Kattouw from comment #2) > Turns out EventLogging's mw.track subscriber is broken, and VisualEditor was > using it for no good reason. I'll port VE to u...