[07:32:50] Analytics, Community-Liaisons (Apr-Jun-2016), User-Johan: Collect information about how we collect user statistics in one place - https://phabricator.wikimedia.org/T132405#2234260 (Qgil) [08:38:10] hi team! [09:03:02] Hi mforns :) [09:11:32] hello joal :] [09:16:42] How are you mforns ? [09:16:49] good thx, you? [09:16:56] not bad :) [09:28:28] Analytics, Analytics-Kanban, Pageviews-API, RESTBase-Cassandra: Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#2234424 (JAllemandou) [09:28:37] Analytics-Kanban, RESTBase-Cassandra: Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#2234426 (JAllemandou) [09:29:09] Analytics-Kanban, RESTBase-Cassandra: Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#1952692 (JAllemandou) Removing pageview-API tag to prevent having automatic Analytics tagging (this belongs to Analytics-Kanban now) [09:53:53] aloha :) [10:00:25] just a quick reminder that today I am off (public holiday) but since you dear ops are both slacking today I'll double check in the evening if you'll need anything to get unblocked :) [10:14:04] elukey, have a nice day! [10:14:05] Thanks elukey, enjoy your day off :) [11:42:06] Analytics-Kanban: Make the AQS unique-devices endpoint return 'devices' as a numeric value, not a string - https://phabricator.wikimedia.org/T133527#2234672 (mforns) [13:21:35] halfak: o/ ? [13:34:01] Hi mforns [13:35:14] thanks for having spotted the AQS uniques bug ! [13:35:21] I had not noticed at all [14:05:08] (PS1) Joal: Correct unique devices endpoint value format [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 [14:05:58] Analytics-Kanban: Make the AQS unique-devices endpoint return 'devices' as a numeric value, not a string - https://phabricator.wikimedia.org/T133527#2235113 (JAllemandou) a:JAllemandou [14:06:54] Analytics-Cluster, Analytics-Kanban: Standardize use of refinery_path over oozie_path in all refinery oozie property files - https://phabricator.wikimedia.org/T133206#2235119 (JAllemandou) [14:07:07] Analytics-Cluster, Analytics-Kanban: Standardize use of refinery_path over oozie_path in all refinery oozie property files - https://phabricator.wikimedia.org/T133206#2225048 (JAllemandou) a:JAllemandou [14:23:30] joal, do you need a CR? [14:23:38] mforns: I do ;) [14:24:00] mforns: corrected the bug you found [14:24:15] joal, I saw it's in my gerrit [14:24:38] I also added mobrovac to let him know about the actually non-working unit testing of int/string stuff [14:27:54] aha [14:28:03] joal, what do you mean with 'v' column? [14:28:08] joal: that's related to deepEqual [14:28:22] re the test [14:28:39] assert.deepEqual(1, '1') == true unfortunately [14:30:00] (CR) Mforns: Correct unique devices endpoint value format (1 comment) [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 (owner: Joal) [14:31:00] also, joal, mobrovac, is this slash normal? '/9007199254740991' [14:31:11] Analytics-EventLogging, Operations, Performance-Team, Patch-For-Review: "Throughput of EventLogging NavigationTiming events" UNKNOWN - https://phabricator.wikimedia.org/T132770#2235182 (fgiunchedi) looks like the alert itself now is no longer unknown, @Krinkle anything else to do within the task? [14:31:37] yes mforns, it's part of the uri [14:31:46] ok ok [14:32:51] (CR) Mforns: Correct unique devices endpoint value format (1 comment) [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 (owner: Joal) [14:46:46] Quarry: Various assortment of improvements - https://phabricator.wikimedia.org/T133545#2235207 (Ankit-Maity) [14:47:01] (CR) Mforns: "mobrovac says the test failing is related to deepequal." (2 comments) [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 (owner: Joal) [14:58:17] (CR) Mforns: [C: 2 V: 2] "Passed the unit tests, and tested in the browser." (2 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/278395 (https://phabricator.wikimedia.org/T131547) (owner: Jdlrobson) [15:14:43] mobrovac: currently trying with deepStrictEqual, but doesn't seem to help :( [15:14:48] mobrovac: any idea? [15:17:12] joal: indeed, we should use that, assert.deepStrictEqual(1, '1'); throws for me [15:20:54] mobrovac: it throws for me as well in test, but when trying to include it in code, doesn't :( [15:23:02] joal: can you create a patch for that so i can take a look at it? [15:23:30] sure mbrovac, uploading a new one on the current change in a minute [15:25:58] (PS2) Joal: Correct unique devices endpoint value format [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 [15:26:11] mobrovac: --^ I expect this to actually fail in test [15:26:21] mobrovac: but doesn't :( [15:28:52] Hey, I'm getting 502s on browser-reports.wmflabs.org – known issue? [15:36:06] James_F, thanks looking [15:36:13] Thanks, mforns! [15:47:16] halfak: \o (trying the other hand!) [15:48:18] o/ joal [15:49:40] sorry to bother halfak, I'm trying to get to some info fast, and you're the best place I know for those [15:50:24] James_F, in the meantime you can use: https://browser-reports.wmflabs.org [15:51:09] wait [15:51:36] James_F, OK seems to work [15:52:14] (CR) Nuria: Allow filtering of data breakdowns (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/278395 (https://phabricator.wikimedia.org/T131547) (owner: Jdlrobson) [15:52:17] mforns: Fixed? Nice. Thanks. [15:52:22] joal, fire away! [15:52:32] :) [15:52:42] James_F, not really, it came back by itself [15:52:50] halfak: it's about getting edit data from mysql, so not sure IRC would be best [15:52:59] halfak: seen my email? [15:53:28] mforns: will be deploying vital signs [15:53:37] nuria_, OK [15:54:43] halfak: was expecting to be able to discuss that during last week meeting, but for once it was me waiting ;) [15:59:59] nuria_: I'm going up to office. Will join standup in a minute [16:00:01] Sorry about missing the last live sync meeting. I was on vacation. :\ [16:00:25] halfak: no bother, as I said, for once, it was on me ;) [16:00:27] a-team: standddupppp [16:00:35] * halfak has to run to a meeting [16:00:37] back later [16:00:50] halfak: I was wondering if you were on holidays or conf or something :) [16:00:56] halfak: later: ) [16:01:10] mforns: standduppp [16:02:10] madhuvishy: hola [16:15:41] Analytics, Commons, Multimedia, Wikidata, and 3 others: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2235673 (matmarex) >>! In T120452#2232451, @Yurik wrote: > I found [[ https://commons.wikimedia.org/wiki... [16:27:51] (PS1) Joal: Improve XML-JSON input format and add mapreduce [analytics/wikihadoop] - https://gerrit.wikimedia.org/r/285203 [16:31:51] Analytics-Kanban: Build a javascript client for the unique devices API - https://phabricator.wikimedia.org/T133159#2235749 (Nuria) [16:31:53] Analytics-Kanban: Unique Devices javascript node module - https://phabricator.wikimedia.org/T133184#2235750 (Nuria) [16:34:06] Analytics-Kanban: Unique Devices javascript node module - https://phabricator.wikimedia.org/T133184#2235760 (Nuria) a:mforns [16:35:07] Analytics: fundraising-tech request: browser version breakdown by country - https://phabricator.wikimedia.org/T133407#2235767 (Nuria) [16:37:11] Analytics, Analytics-Dashiki, Easy: Dashiki breakdown layout problems. UI - https://phabricator.wikimedia.org/T133312#2235770 (Nuria) [16:39:50] joal: have you tried printing out with JSON.stringify() in https://gerrit.wikimedia.org/r/#/c/285180/2/test/features/unique-devices/unique-devices.js right before line 81 to make sure you actually get strings? [16:40:05] mo [16:40:12] mobrovac: didn't, will do [16:40:23] mobrovac: after meeting thouhg :) [16:41:26] mobrovac: thanks for pointer [16:46:27] Analytics: fundraising-tech request: browser version breakdown by country - https://phabricator.wikimedia.org/T133407#2235821 (DStrine) @Nuria Thanks for the browser reports are interesting. However the country breakdown would help a bit more for the particular questions we are trying to answer. I've neve... [16:48:18] Analytics, Commons, Multimedia, Wikidata, and 3 others: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2235826 (Yurik) JsonConfig can easily allow us to store all allowed licenses as a "config" page - a JSON... [16:50:45] Analytics, Pageviews-API: Allow for arbitrary date ranges in /top endpoint of pageviews API - https://phabricator.wikimedia.org/T133176#2224280 (Nuria) This is computationally intensive and too big for us to generate at this time. Declining. [16:50:51] Analytics, Pageviews-API: Allow for arbitrary date ranges in /top endpoint of pageviews API - https://phabricator.wikimedia.org/T133176#2235851 (Nuria) Open>declined [16:52:21] Analytics, Services: Get http level ops stats for AQS from varnish - https://phabricator.wikimedia.org/T133171#2224202 (Nuria) [16:54:28] ottomata: coming in to office today? [17:01:07] a-team, today I worked earlier, will log off in an hour [17:02:05] Analytics, Commons, Multimedia, Wikidata, and 3 others: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2235937 (Yurik) Another alternative is to actually reuse `.tabular` for storing this data, instead of cr... [17:12:11] Analytics, Services: Get http level ops stats for AQS from varnish - https://phabricator.wikimedia.org/T133171#2235981 (GWicke) Unless something has changed, I believe all requests to AQS traverse Varnish and RESTBase. This means that the RB metrics already cover all Varnish cache misses. While the Varni... [17:13:12] mobrovac: interesting finding when testing a bit deeper: different test results for different backends [17:13:44] mobrovac: 'test/utils/run_tests.sh test all' fails with cassandra but succeeds with sqllite [17:13:57] mobrovac: related to data types [17:14:44] hm [17:15:11] I console logged as suggested: when using cassandra, strings, when using sqllite, numbers [17:15:28] and you send them as strings in json? [17:15:39] hm? [17:16:42] joal, just scheduled a meeting for us tomorrow in my morning (your afternoon) [17:16:47] I hope that'll work [17:16:50] running off to lunch now [17:16:52] o/ [17:17:01] Thanks a lot halfak|Lunch :) [17:17:40] mobrovac: insertion is done using yaml endpoint definition, so I don't really know what format is swent [17:17:58] hm hm [17:18:17] ok, obviously we'll need to dig a bit deeper there to get to the bottom of this [17:18:24] I suggst not doing it now :P [17:18:33] mobrovac: Right :) [17:18:44] mobrovac: when would it be good for you? [17:19:25] i can try to squeeze it in tmrrw afternoon [17:20:10] mobrovac: look at my calendar, tomorrow afternoon is busy but if we find a match, I'll go for it :) [17:20:23] kk :) [17:23:41] Analytics, RESTBase: REST API entry point web request statistics at the Varnish level - https://phabricator.wikimedia.org/T122245#1899324 (Nuria) @GWicke varnish should be reporting this stats , not the cluster, correct? Otherwise stats reporting might take hours after requests have happened (cluster pr... [17:26:46] Analytics, RESTBase: REST API entry point web request statistics at the Varnish level - https://phabricator.wikimedia.org/T122245#2236028 (GWicke) @Nuria, these metrics would be derived from log data in an asynchronous fashion. We already have some hive-based stats for accesses to the REST API in general... [17:33:32] Analytics, RESTBase: REST API entry point web request statistics at the Varnish level - https://phabricator.wikimedia.org/T122245#2236036 (Nuria) @Gwicke: I think these stats if they are to be used by monitoring should be real time(ish) and if possible sent by varnish itself via statsd, What is the ratio... [17:42:24] Analytics, Pageviews-API: Allow for arbitrary date ranges in /top endpoint of pageviews API - https://phabricator.wikimedia.org/T133176#2236122 (MusikAnimal) Bummer. Thanks anyway! [17:42:47] Analytics, RESTBase: REST API entry point web request statistics at the Varnish level - https://phabricator.wikimedia.org/T122245#2236124 (GWicke) > What is the rationale to use the cluster to compute them if we have a statsd client for varnish? I am not aware of any general real-time traffic analytics... [17:43:44] !log deployed new vitalsigns code to https://vital-signs.wmflabs.org [17:43:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [17:43:56] Analytics, Services: Get http level ops stats for AQS from varnish - https://phabricator.wikimedia.org/T133171#2224202 (madhuvishy) @GWicke Do cache hits show up in request logs? [17:46:04] Analytics, Services: Get http level ops stats for AQS from varnish - https://phabricator.wikimedia.org/T133171#2236150 (GWicke) > Do cache hits show up in request logs? Yes, logs cover all accesses, hit or miss. [17:54:51] nuria_: hey! regarding https://phabricator.wikimedia.org/T133176, do you think it would be feasible/worthwhile to cap a start/end date range at 30 days? [17:54:53] Currently you can add `all-days` as the value for `month`, so would an arbitrary range within those 30 days be any more intensive? [17:56:52] MusikAnimal: the problem is that all of this data is precomputed on hive and stored in cassandra [17:57:25] cassandra essentially already has all these results for a month - and just gives it to you [17:57:47] I see [17:58:28] joal, I've been looking to aqs errors wwithout success until now [17:58:30] well what about precomputing weekly stats for the year? so you could just give it the `year` and `week`, leaving `month` and `day` empty [17:58:34] will continue tomorrow! [17:58:53] MusikAnimal: we will have that when we have a years data i think [17:59:02] until now we don't [17:59:39] madhuvishy: you mean once you have a years worth of data, you could break it down week by week? sorry was't sure what you meant [18:00:17] see you tomorrow a-team! bye [18:00:50] joal: ^ if you have anything more to add for MusikAnimal's question [18:01:05] To be clear this request is mostly for the benefit of The Signpost. They are currently using the less accurate data dumps for the weekly traffic report [18:02:28] MusikAnimal: like weekly stats for top? [18:02:43] indeed! [18:03:31] hmmm that sounds doable [18:03:44] awesome! :) [18:04:19] The Signpost would be most pleased [18:04:22] although i don't know what the team feels - can you file a task so we can talk about it in the next meeting may be tomorrow [18:04:38] sure, there was this one but I can make a new one https://phabricator.wikimedia.org/T133176 [18:04:49] thank you very much! :D [18:05:56] yeah just make a new one - even if we did it it would take a while I suppose - especially to compute and backfill data. also Dan, who works on the API a lot is on vacation so apologies in advance if there are some delays in getting answers [18:07:04] no worries, thank you nonetheless; this is purely a nice-to-have and are certainly willing to wait :) [18:08:15] no problem! and thanks for all your work in making awesome tools :) [18:08:24] my pleasure [18:14:40] Analytics, Pageviews-API: Provide weekly top pageviews stats - https://phabricator.wikimedia.org/T133575#2236232 (MusikAnimal) [18:33:04] MusikAnimal: reading backlog now [18:33:37] MusikAnimal: your request is to calculate weekly top data? [18:34:04] MusikAnimal: top1000 per week? [18:34:39] yeah [18:34:49] precisely! but actually even top 100 would suffice [18:35:15] MusikAnimal: storage wise this would be about 1000 (projects) *1000 (articles) *50 (weeks)~1.000.000 [18:35:37] is that an issue? :-/ [18:35:45] MusikAnimal: so we would need 1 million more rows annually which seems tenable from space side [18:35:52] MusikAnimal: well, more storage is more $$ [18:36:09] MusikAnimal: but in this case space doesn't seem to be an issue (joal to confirm) [18:36:17] (PS1) Madhuvishy: Fix data breakdowns selectors not visible on the VitalSigns UI [analytics/dashiki] - https://gerrit.wikimedia.org/r/285223 (https://phabricator.wikimedia.org/T133312) [18:36:46] sure, I thought storage was cheap, but I've never been the one who has to pay for it =P [18:37:22] MusikAnimal: think that if you want a breakdown by user /bot or mobile desktop [18:37:53] MusikAnimal: you are adding columns and will be increasing storage as your dimensions increase [18:37:59] yeah ideally we'd have that oo [18:38:19] nuria_: the data breakdowns thing kept bugging me so i patched, feel free to check if it fixes the problem and merge! [18:38:52] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Puppetize and make useable confluent kafka packages - https://phabricator.wikimedia.org/T132631#2236286 (Ottomata) For my own reference, a few defaults have changed in Kafka's server.properties.erb for 0.9. I'm making these changes to the defaul... [18:38:57] MusikAnimal: did you tested on safari/ff? [18:39:08] sorry madhuvishy did you tested on ff safari? [18:39:09] nuria_: you mean me? [18:39:10] yes [18:39:12] i did [18:39:13] jaja [18:39:16] haha [18:39:17] YES! sorry [18:39:37] MusikAnimal: so any breakdown increases size as you have that data all over again, makes sense? [18:39:43] yeah definitely [18:40:33] MusikAnimal: so storage ends up being too large for a cluster of three machines. Now, joal to confirm but weekly might be ok storage wise , computation wise i just do not know if it is doable. *seems* that it shoudl be too [18:40:38] *should [18:42:13] madhuvishy: that fixes just colors, correct? [18:42:38] nuria_: well the grey panel on the left wasn't extending all the way to the bottom [18:42:57] i think the browsers assume that the bottom is somehow the bottom of the right column [18:43:03] i dont know why [18:43:16] removing top:0, bottom:0 seemed to fix it [18:43:56] madhuvishy: ah ok, we can leave ticket open as line height on that component is also off and that is why it extends so much below [18:44:04] madhuvishy: let me test & merge [18:44:22] the line height shouldn't matter though [18:44:26] if i added 100 projects [18:44:33] i will get a long component [18:44:42] nuria_: ^ [18:48:00] Quarry: Quarry shows old username after user was renamed - https://phabricator.wikimedia.org/T73064#771640 (Stefan2) >>! In T73064#2001109, @Krenair wrote: > * ... now we have two As? The username field is supposed to be unique according to tables.sql. This may also happen even if you do not implement user r... [19:25:46] madhuvishy: yes, the line height of the breakdown is the one i was thinking of [19:30:21] madhuvishy: fyi, remember that research that yuvi and oliver were going to do regarding IPs? We worked on a plan and this is what will happen: https://meta.wikimedia.org/wiki/Research:Mobile_User_Behavioural_Differences [19:34:23] madhuvishy: tested in ff and chrome, my safari is old and doesn't support promise so things do not work anyways [19:35:03] (CR) Nuria: [C: 2 V: 2] "Tested in FF & chrome, fixes issues with background color of breakdown component." [analytics/dashiki] - https://gerrit.wikimedia.org/r/285223 (https://phabricator.wikimedia.org/T133312) (owner: Madhuvishy) [19:47:58] nuria_: cool [19:48:11] thanks, should I deploy to staging? [19:54:42] madhuvishy: iam deploying & testing now [19:54:43] np [19:55:16] !log deployed new vitalsigns code to https://vital-signs.wmflabs.org [19:55:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [19:55:48] nuria_: cool! [20:36:18] (PS1) Nuria: [WIP] Add out of service banner to dashiki [analytics/dashiki] - https://gerrit.wikimedia.org/r/285255