[00:02:23] Analytics-Dashiki: Dashiki needs to have a friendlier mobile view. - https://phabricator.wikimedia.org/T75030#832461 (Nuria) Since the only thing visible in mobile is going to be -pretty much - the graph itself it should adapt to viewport sizes of different mobile devices. Please see attached screenshots to... [00:04:28] Analytics, Multimedia, MediaWiki-extensions-UploadWizard: Collect data on which unsupported file formats users are trying to upload - https://phabricator.wikimedia.org/T77796#832476 (Gilles) [00:09:04] bmansurov: https://wikitech.wikimedia.org/wiki/Analytics/Dashboards [00:09:27] nuria__: thanks [00:10:22] kevinator: ahem.... is there a way to add screenshots to phabricator [00:10:50] Analytics-Wikimetrics, Analytics-Engineering: User reads result of validation after creating a cohort - https://phabricator.wikimedia.org/T76914#832500 (kevinator) [00:12:16] nuria__: Drag the image into the editor. [00:12:22] nuria__: It Just Works™. [00:12:27] James_F: ooohhhh [00:12:49] James_F: see what happens when you use bugzilla too long... [00:13:02] nuria__: :-) [00:13:10] Analytics-Dashiki: Dashiki needs to have a friendlier mobile view. - https://phabricator.wikimedia.org/T75030#832507 (Nuria) {F20122} {F20123} {F20124} [00:18:24] Analytics-Wikimetrics, Analytics-Engineering: [Dev - 8 pts] upgrade wikimetrics to trusty (using same install process as we have now. Probably just setting up new instances, not uprgrading them) - https://phabricator.wikimedia.org/T76769#832515 (kevinator) [00:19:19] Analytics-Wikimetrics, Analytics-Engineering: [Dev 13 pts] change wikimetrics in labs to use debian packages, try in staging 1st, make sure this works alongside of pip install with webserver part, too - https://phabricator.wikimedia.org/T76770#832518 (kevinator) [00:20:25] Analytics-Wikimetrics, Analytics-Engineering: RAE implemented using Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T76441#832521 (kevinator) [00:20:31] Analytics-Wikimetrics, Analytics-Engineering: Epic: WikimetricsUser has all metrics reimplemented with Data Warehouse - https://phabricator.wikimedia.org/T76387#832522 (kevinator) [00:21:51] Analytics-Wikimetrics, Analytics-Engineering: Story: WikimetricsUser deletes user from cohort [21 pts] - https://phabricator.wikimedia.org/T75350#832529 (kevinator) [00:22:25] Analytics-Engineering, Analytics-Wikimetrics: Story: WikimetricsUser reports pages edited by cohort [13pts] - https://phabricator.wikimedia.org/T75072#832531 (kevinator) p:Normal>High [00:23:08] Analytics-Wikimetrics, Analytics-Engineering: Story: WikimetricsUser reads user names in a JSON report [8 pts] - https://phabricator.wikimedia.org/T74747#832534 (kevinator) p:Low>Volunteer? [00:23:28] Analytics-Wikimetrics, Analytics-Engineering: Story: Wikimetrics compiles target-site breakdown for remaining metrics [34 pts] - https://phabricator.wikimedia.org/T74738#832535 (kevinator) [00:24:51] Analytics-Wikimetrics: Puppet is restarting queue every time it runs - https://phabricator.wikimedia.org/T74565#832541 (kevinator) [00:24:58] Analytics-Wikimetrics: Tasks should clean up after themselves when they fail in STARTING phase - https://phabricator.wikimedia.org/T74563#832542 (kevinator) [00:26:42] Analytics-Wikimetrics: labsdb issues forced wikimetrics-scheduler to be stopped - https://phabricator.wikimedia.org/T74281#832543 (kevinator) @milimetric can we close this? [00:27:12] Analytics-Wikimetrics: labsdb issues forced wikimetrics-scheduler to be stopped - https://phabricator.wikimedia.org/T74281#832544 (kevinator) [00:27:56] Analytics-Wikimetrics: Story: VSUser has data from 2008 to now - https://phabricator.wikimedia.org/T74133#832547 (kevinator) [00:28:33] Analytics-Wikimetrics: Scheduling reports for the future - https://phabricator.wikimedia.org/T74119#832548 (kevinator) [00:29:21] Analytics-Wikimetrics: Give the option of using the same time range for all reports for a given cohort - https://phabricator.wikimedia.org/T74117#832551 (kevinator) [00:30:29] Analytics-Wikimetrics: Story: WikimetricsUser is able to chain metrics - https://phabricator.wikimedia.org/T73614#832555 (kevinator) [00:32:27] Analytics-Wikimetrics: slow report result and report list fetching - https://phabricator.wikimedia.org/T73603#832557 (kevinator) [00:35:39] Analytics-Wikimetrics: Story: AnalyticsEng has productionized wikimetrics and limn servers - https://phabricator.wikimedia.org/T73455#832562 (kevinator) [00:35:41] Analytics-Wikimetrics, Analytics-Engineering: EPIC: Productionizing Wikimetrics - https://phabricator.wikimedia.org/T76726#832563 (kevinator) [00:36:05] Analytics-Wikimetrics: 1st run of recurrent report has empty resultset - https://phabricator.wikimedia.org/T73338#832565 (kevinator) [00:36:24] Analytics-Wikimetrics: Story: AnalyticsEng uses connection pooling on database URL - https://phabricator.wikimedia.org/T73140#832566 (kevinator) [00:36:41] Analytics-Wikimetrics: Add MAILTO to backup crontab on puppet to be notified of failures - https://phabricator.wikimedia.org/T72853#832568 (kevinator) [00:37:13] Analytics-Wikimetrics: Improve perf of rolling active editors with dba's suggestions - https://phabricator.wikimedia.org/T71610#832571 (kevinator) [00:38:10] Analytics-Wikimetrics: Story:d WikimetricsUser runs 'Rolling Recurring old active editors' report - https://phabricator.wikimedia.org/T71569#832573 (kevinator) [00:38:30] Analytics-Wikimetrics: Separate concurrent report queue and regular 'default' queue - https://phabricator.wikimedia.org/T71523#832576 (kevinator) [00:38:39] (PS1) Bmansurov: Update click tracking graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/178381 [02:06:00] (PS3) Unicodesnowman: Make Dashiki responsive [analytics/dashiki] - https://gerrit.wikimedia.org/r/177487 [02:10:35] Analytics-Dashiki: Dashiki needs to have a friendlier mobile view. - https://phabricator.wikimedia.org/T75030#832668 (Unicodesnowman) I've uploaded a new patch which will make the graph fill the viewport. It doesn't look too good in portrait -- the axis labels overlap, and the fidelity is too low. I think it... [02:17:47] (CR) Nuria: "Looks a lot better, thank you. Since dan was doing the initial CR I will let him do the merge if proceeds." [analytics/dashiki] - https://gerrit.wikimedia.org/r/177487 (owner: Unicodesnowman) [02:46:27] Krinkle: i just looked at ESI... juas! [02:46:44] Krinkle: it's like the old server-side-include i used to use 10 years ago [02:46:55] nuria__: Yeah, kind of. [02:46:57] but at the caching layer! [02:47:18] Krinkle: like fashion, everything comes back [02:47:51] nuria__: It's quite powerful. In this case it suddenly manifests as a very modern and desired pattern in a properly modular application. Composing different componenets together in one. [02:48:20] But not for header/footer, but e.g. separating static information from personalised content. [02:48:36] which, .... yeah, it's just the same. [02:48:51] Krinkle: no, i get it, we used to do that in Amazon back in the day, all home grown [12:31:16] (PS1) Yurik: Generate monthly data [analytics/zero-sms] - https://gerrit.wikimedia.org/r/178479 [12:34:56] (PS2) Yurik: Generate monthly data [analytics/zero-sms] - https://gerrit.wikimedia.org/r/178479 [12:35:54] (PS3) Yurik: Generate monthly data [analytics/zero-sms] - https://gerrit.wikimedia.org/r/178479 [12:36:17] (CR) Yurik: [C: 2 V: 2] Generate monthly data [analytics/zero-sms] - https://gerrit.wikimedia.org/r/178479 (owner: Yurik) [13:35:20] Analytics-Refinery: cp3003.esams.wikimedia.org lost a kafka message on 2014-12-09T00:38:11 - https://phabricator.wikimedia.org/T77989 (QChris) NEW p:Triage [13:36:11] !log Marked raw upload webrequest partition for 2014-12-09T00/1H ok (See {{PhabT|77989}}) [14:24:44] (PS1) Yurik: Fixed bug in weblogs2 - tuple instead of lines [analytics/zero-sms] - https://gerrit.wikimedia.org/r/178500 [14:25:09] (CR) Yurik: [C: 2 V: 2] Fixed bug in weblogs2 - tuple instead of lines [analytics/zero-sms] - https://gerrit.wikimedia.org/r/178500 (owner: Yurik) [14:36:07] (PS1) Yurik: Proper list sorting [analytics/zero-sms] - https://gerrit.wikimedia.org/r/178503 [14:36:36] (CR) Yurik: [C: 2 V: 2] Proper list sorting [analytics/zero-sms] - https://gerrit.wikimedia.org/r/178503 (owner: Yurik) [14:41:28] Analytics-Refinery: cp3005.esams.wikimedia.org lost a kafka message on 2014-12-08T15:46:50 - https://phabricator.wikimedia.org/T77999 (QChris) NEW p:Triage [14:42:14] !log Marked raw upload webrequest partition for 2014-12-08T15/1H ok (See {{PhabT|77999}}) [14:44:15] Analytics-Refinery: cp3005.esams.wikimedia.org lost a kafka message on 2014-12-08T18:22:43. - https://phabricator.wikimedia.org/T78000 (QChris) NEW p:Triage [14:45:20] !log Marked raw upload webrequest partition for 2014-12-08T18/1H ok (See {{PhabT|78000}}) [15:29:17] milimetric, sorry I had started hexchat but not logged to the channel, my bad [15:29:28] anyone aware of eventlogging issues? [15:30:04] not that i heard [15:30:25] It seems we're net getting events into the database :-( [15:30:50] the edit team's dashboards has data on the 9th [15:30:58] and i could swear the numbers went up in the last hour [15:31:09] but i'll double check after / during staff [15:31:19] Mhmm. k. [15:32:18] milimetric: Google does not let me in. Can you join the hangout? [15:35:25] (PS4) Milimetric: Make Dashiki responsive [analytics/dashiki] - https://gerrit.wikimedia.org/r/177487 (owner: Unicodesnowman) [15:35:41] (CR) Milimetric: [C: 2 V: 2] "Thank you very much unicodesnowman!" [analytics/dashiki] - https://gerrit.wikimedia.org/r/177487 (owner: Unicodesnowman) [15:36:44] nuria__: merged unicodesnowman's change after a clean rebase [15:37:08] milimetric: got it, will deploy latest build to staging [15:37:27] 1st thing after current meeting [15:42:37] nuria__: Jan 1st 2014 is 1388552400000 if you're gonna uncomment that line [15:44:03] EventLogging seems fine. The events are in the m2 master. [15:44:55] milimetric: k [15:56:12] jgage: and qchris, want to just do our sync up now? [15:56:19] fine with me [15:56:27] ottomata, jgage: fine by me too :-) [15:56:47] cool ok, but i guess batcave is occupied [15:56:57] The invite has a hangout. [15:57:02] But it does not let me in :-/ [15:57:03] oh we have our own link [15:57:31] https://plus.google.com/hangouts/_/wikimedia.org/aotto-jgerard [16:11:22] Analytics-Wikimetrics: EPIC: Longterm fix of data warehouse replicating to Labs - https://phabricator.wikimedia.org/T78009 (ggellerman) NEW p:Triage [16:12:56] Analytics-Refinery: Getting Ananth started - https://phabricator.wikimedia.org/T77196#833711 (Tnegrin) [16:13:21] (PS1) Nuria: Release 2014-12-09 [analytics/dashiki] - https://gerrit.wikimedia.org/r/178536 [16:14:20] (CR) Nuria: [C: 2 V: 2] "Self-merging release per team's policy." [analytics/dashiki] - https://gerrit.wikimedia.org/r/178536 (owner: Nuria) [16:31:15] milimetric, mforns_lunch the meta content needs to be served under https if we want dashiki to be able to show it under https [16:31:33] milimetric, mforns_lunch: i will do some changes to that extent [16:33:40] Analytics-Wikimetrics, Analytics-Engineering: change wikimetrics in labs to use debian packages, try in staging 1st, make sure this works alongside of pip install with webserver part, too [13 pts] - https://phabricator.wikimedia.org/T76770#833740 (ggellerman) [16:34:38] nuria__: thanks! i've gotta grab lunch and prepare for the interview, but let me know if I can help [16:35:03] milimetric: no owrries, i will do in-place changes for showcase and send couple gerrit changes along [16:35:26] *no worries [16:57:59] milimetric mforns_lunch nuria__ : here is the deck for today’s showcase [16:58:01] https://docs.google.com/presentation/d/1LmkWEpcJD0-AtQMRmLEFSM-T_9hNCWVGvCvLLYIrkBQ/edit?usp=sharing [17:01:52] nuria__, I see, do you want me to do it? [17:06:01] mforns: no, np, i will do it [17:06:13] nuria__: ok, thanks [17:08:28] Analytics-Engineering: Generate.py needs to write to a separate history.json file per configured instance - https://phabricator.wikimedia.org/T77936#833802 (kevinator) p:Triage>Normal [17:10:35] Analytics-Engineering: [Volunteer] Generate.py needs to write to a separate history.json file per configured instance - https://phabricator.wikimedia.org/T77936#833807 (kevinator) [17:11:09] Analytics-Engineering: [Volunteer] Improve Generate.py [13 pts for the Analytics Eng team] - https://phabricator.wikimedia.org/T76407#833808 (kevinator) [17:11:34] (PS1) Nuria: Release 2014-12-09 (corrected breakdown component) [analytics/dashiki] - https://gerrit.wikimedia.org/r/178539 [17:15:56] (CR) Nuria: [C: 2 V: 2] "Self merging release per team's policy." [analytics/dashiki] - https://gerrit.wikimedia.org/r/178539 (owner: Nuria) [17:17:23] mforns, milimetric : dashiki deployed to staging (corrected couple issues, one is already merged to master the other I corrected in place, will be sending a changeset for mediawiki's storage bower component) [17:17:27] https://metrics-staging.wmflabs.org/static/public/dash/#projects=ruwiki,itwiki,eswiki,jawiki/metrics=DailyPageviews [17:17:59] cool [17:18:39] i think that makes the breakdowns a little more visible [17:18:51] and it's nice to see the jump is explained by the inclusion of mobile [17:19:15] Analytics-Engineering: EPIC EL documentation & evangelism - https://phabricator.wikimedia.org/T76795#833856 (ggellerman) Set up meeting for Fri Dec 11, 2014 for Kevin, Grace, Dan, Nuria and Toby to discuss plan [17:22:06] Analytics-Engineering: EPIC EL documentation & evangelism - https://phabricator.wikimedia.org/T76795#833876 (kevinator) p:Triage>Normal [17:22:53] Analytics-Engineering: set up weekly office hours for EL - https://phabricator.wikimedia.org/T76796#833895 (ggellerman) p:Triage>Normal [17:23:02] Analytics-Engineering: think through capacity management & Information Architecture - https://phabricator.wikimedia.org/T76803#833897 (kevinator) p:Triage>Normal [17:23:15] Analytics-Engineering: drive consistency across all of the schemas (standard fields & def'ns) - https://phabricator.wikimedia.org/T76801#833900 (kevinator) p:Triage>Normal [17:23:24] Analytics-Engineering: work with Rachel on how best to evangelize - https://phabricator.wikimedia.org/T76797#833904 (ggellerman) p:Triage>Normal [17:23:27] Analytics-Engineering: set up hackaton at MWDS - https://phabricator.wikimedia.org/T76800#833905 (kevinator) p:Triage>Normal [17:23:51] Analytics-Engineering: set up mtg w/Visual Editor (Ron) to talk about what we did months back - https://phabricator.wikimedia.org/T76798#833908 (ggellerman) [17:25:00] kevinator: could we get edit permits on the showcase deck [17:25:09] Analytics-Engineering: Getting Started in wikimetrics doc - https://phabricator.wikimedia.org/T77075#833927 (kevinator) [17:26:23] Analytics-Engineering: Community has a developer doc "Getting Started with Wikimetrics" - https://phabricator.wikimedia.org/T77075#833934 (kevinator) [17:26:42] Analytics-Engineering: Community has a developer doc "Getting Started with Wikimetrics" - https://phabricator.wikimedia.org/T77075#825421 (kevinator) [17:29:29] Analytics-Engineering: Evaluate UX of Gerrit documentation - https://phabricator.wikimedia.org/T77059#833944 (kevinator) [17:29:43] Analytics-Engineering: Evaluate UX of Gerrit documentation - https://phabricator.wikimedia.org/T77059#833945 (kevinator) p:Triage>Low [17:30:39] Analytics-Engineering: Evaluate UX of Gerrit documentation - https://phabricator.wikimedia.org/T77059#825309 (kevinator) [17:38:16] milimetric, ping [17:38:47] hi rtnpro [17:38:47] milimetric, regarding the logger integration, I had some questions [17:38:50] yes [17:39:20] milimetric, so, the logging conf can go as a dict in a python file or in the YAML file as well [17:40:04] mforns, milimetric: wikimetrics deployed to staging, you can see timezone picker here: https://metrics-staging.wmflabs.org/reports/create/ [17:40:11] milimetric, and also, we can expose some parameters in the command line option to override the default logging config [17:40:14] mi [17:40:17] nuria__: cool :) [17:40:26] nuria__, thanks! [17:40:52] milimetric, now, the question is that what options do we want to expose to the command line or the YAML config files [17:40:52] rtnpro: so the yaml file will live separately from this generate.py logic [17:40:56] we just haven't separated it yet [17:41:13] as far as options, I'd have to take a closer look. What options are available? [17:42:39] milimetric, quite a few, see here https://github.com/rtnpro/analytics-limn-mobile-data/blob/logging/logger.py [17:43:16] milimetric, it's a nested dictionary, so you can configure a lot of things [17:43:32] milimetric, I'd suggest keeping things simple [17:44:18] milimetric, we can only allow configuring the log file path, and the host and port for logstash backend [17:44:29] yeah, i just want to pick the minimal options so that we can configure this easily via puppet [17:44:47] if there are two or three options, we can keep it command line, and if there are more, we can make a separate config file [17:44:49] milimetric, does it sound good? [17:45:43] milimetric, a config file sounds better +1, will keep it simple and flexible [17:45:54] rtnpro: looks good, actually, let's just expose one parameter for now - the level of logging [17:45:58] and default to DEBUG [17:46:22] you can expose that on the command line, simple enough [17:46:32] milimetric, ok [17:46:38] the logstash parameters would be more complicated no>? I have no idea how logstash works :) [17:47:26] milimetric, we just need to specify the host and the port in the logging configuration, and python-logstash takes care of the rest [17:49:56] (PS1) Nuria: Consolidating our css styles to one file [analytics/dashiki] - https://gerrit.wikimedia.org/r/178547 [17:49:58] rtnpro: cool, then exposing those would be good, and probably at that point you should put the config in a separate file [17:50:16] rtnpro: i've got an interview in 10 min. so I'll leave it at that, but everything looks good, thanks! [17:50:34] milimetric, np [17:50:49] milimetric, I will update the patch and ping you to review :) [17:50:57] cool, thx [17:51:21] oh one more thing: you've been doing the pull request which I think is ok, YuviPanda has a github -> gerrit migration script [17:51:36] is that how you contributed to quarry as well, or did you know / use gerrit? [17:58:35] milimetric: nope, that script is dead [17:58:45] milimetric: but rtnpro has been making patches on gerrit so is all good :) [17:59:13] ah good, rtnpro: then would you mind changing your pull request into a change to the analytics/limn-mobile-data project? [18:00:03] milimetric, I will :) [18:00:23] thanks much [18:00:56] (PS2) Nuria: Consolidating our css styles to one file [analytics/dashiki] - https://gerrit.wikimedia.org/r/178547 [18:02:07] Analytics-Refinery: Geo-coding UDF - https://phabricator.wikimedia.org/T77683#833989 (ggellerman) from Dec 9, 2014 RDStandup: have Reid review code [18:28:56] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016 (kevinator) NEW p:High [18:30:48] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016#834032 (kevinator) a:Ottomata [18:31:49] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016#834044 (Ottomata) Priority high!? [18:32:00] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016#834045 (Ottomata) [18:32:21] Analytics-Dashiki: Story: Dashiki uses Mediawiki for storage [13 pts] - https://phabricator.wikimedia.org/T70448#834046 (kevinator) a:mforns [18:32:50] Analytics-Dashiki: Story: Dashiki uses Mediawiki for storage [13 pts] - https://phabricator.wikimedia.org/T70448#834047 (kevinator) Open>Resolved This is implemented and ready to showcase. [18:36:10] Analytics: Upgrade Analytics Cluster to Trusty, and then to CDH 5.2 - https://phabricator.wikimedia.org/T1200#834049 (Ottomata) Hm, might want to chill on the CDH 5.2 upgrade until this is fixed: https://issues.apache.org/jira/browse/HDFS-7489 [18:36:53] Analytics: Ops has feedback from initial vetting of data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T76480#834050 (kevinator) [18:39:48] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016#834061 (ellery) I'm not sure how to interpret priorities. For context, this week I am fine working with Mahout locally and playing with toy data sets. But it would be great to take a crack at building re... [18:40:08] Analytics-Engineering: EPIC: data warehouse - https://phabricator.wikimedia.org/T76382#834063 (kevinator) [18:43:01] Analytics-Engineering: Eng has vetted data in Data Warehouse - https://phabricator.wikimedia.org/T78019 (kevinator) NEW p:Triage [18:43:24] Analytics-Wikimetrics, Analytics-Engineering: Eng has vetted data in Data Warehouse - https://phabricator.wikimedia.org/T78019#834072 (kevinator) p:Triage>High [18:45:43] Analytics: Ops has feedback from initial vetting of data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T76480#834091 (kevinator) [18:46:07] Analytics, Analytics-Wikimetrics: Ops has feedback from initial vetting of data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T76480#834095 (kevinator) [18:46:50] Analytics, Analytics-Wikimetrics: Ops has feedback from initial vetting of data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T76480#834097 (kevinator) Resolving for @nuria [18:46:52] Analytics-Engineering: EPIC: data warehouse - https://phabricator.wikimedia.org/T76382#834101 (kevinator) [18:47:09] Analytics, Analytics-Wikimetrics: Ops has feedback from initial vetting of data in Data Warehouse [13 pts] - https://phabricator.wikimedia.org/T76480#834102 (kevinator) a:kevinator>Nuria [18:52:30] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016#834113 (Ottomata) Using all revisions, or just current page text? [18:59:42] Analytics-Engineering: EPIC: data warehouse - https://phabricator.wikimedia.org/T76382#834133 (fbstj) [19:00:29] Seems I cannot join the showcase hangout :-( [19:01:23] * qchris_away shakes fist at Google [19:01:34] Now it worked :-) [19:01:45] Only needed like 10 tries, and 1 fist shaking. [19:04:45] I found an etherpad linked from a Phabricator task that has meeting notes with details of hiring. Is that stuff that is supposed to be public? [19:17:33] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016#834162 (ellery) [19:18:02] Analytics-Refinery: Eng uses Mahout installed on Hadoop cluster - https://phabricator.wikimedia.org/T78016#834164 (ellery) I will never need the revisions. [19:27:35] milimetric, nuria__: thanks for the support in the demo :] [19:28:02] Analytics-Engineering: Pageview data files for mobile breakdowns: absence of data should not be represented as 'zero' - https://phabricator.wikimedia.org/T78025 (Nuria) NEW p:Triage [19:33:53] got to go elsewhere, headed to a cafe, back shortlyl [19:34:29] nuria__: Hi, I've created new SQLs (limn-mobile-data) and am trying to generate data for my local testing. Since in SQL files I use new schemas, I get an error while generating data. The error tells me that the table I'm querying doesn't exist. So I was wondering if you could tell me at what stage new schemas will be transformed to new tables? Thank you. [19:34:58] nuria__: related patch: https://gerrit.wikimedia.org/r/#/c/178381/ [19:35:00] bmansurov: not until data appears for them in the pipeline [19:35:09] nuria__: I see [19:35:21] bmansurov: as tables are created by teh incoming stream [19:35:23] *the [19:36:11] nuria__: Thanks! Also, we are splitting the MobileWebClickTracking schema into multiple tables, and I'd like to know if we can still keep querying the old table to get historical data. [19:36:51] bmansurov: yes, tables are not deleted unless you send us a ticket or an e-mail about that [19:37:05] nuria__: great, thank you. [20:41:03] ottomata, can we talk for 15 min? I can send you an invite for later in the day, or if you have time now, we can talk now. [20:42:07] Analytics: Fix Varnishkafka delivery error icinga warning - https://phabricator.wikimedia.org/T76342#834591 (Ottomata) [20:42:18] hm, gimme a few leila, maybe 15 or 30 mins? [20:42:44] in that case, I'll send you an invite for a bit later, since I have another meeting. [20:42:45] thanks! [20:42:58] k [20:43:03] oo, well [20:43:15] i mean, if you can't do later, maybe now is fine, i was about to merge something that could hold me up for a bit [20:43:33] no, please do it. I'm not in that much of a hurry. thanks! [20:43:36] ok [20:47:56] (PS1) Nuria: Change mediawiki path to be a protocol relative url [analytics/mediawiki-storage] - https://gerrit.wikimedia.org/r/178603 [20:49:00] (Abandoned) Nuria: Consolidating our css styles to one file [analytics/dashiki] - https://gerrit.wikimedia.org/r/178547 (owner: Nuria) [20:57:55] qchris: was it me who over-shared on etherpad? [20:59:04] (CR) Mforns: [C: -1] "3 tests are failing :]" [analytics/mediawiki-storage] - https://gerrit.wikimedia.org/r/178603 (owner: Nuria) [21:16:59] (PS2) Bmansurov: Update click tracking graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/178381 [21:18:26] ori, nuria__ : Schema:EventCapsule adds a format: "utc-millisec" or "uuid5-hex" to some fields. Is this validated? Is it documented anywhere? [21:19:34] spagewmf: adds? capsule has not changed in almost a year [21:20:08] spagewmf: maybe i missunderstood, is your question how is this calculated? [21:20:38] nuria__: me and EL go back 2 years :) , so it's new to me. I'm wondering what the permissible values are, whether it does anything. [21:21:22] spagewmf: that is calculated by EL client so you do not need to send it yourself, the client will wrap your event in a capsule [21:21:27] that will contain those values [21:21:35] spagewmf: makes sense? [21:23:17] nuria__: I mean when writing a schema, if we want to say what the format of some field is beyond just its type, can we add a format key? [21:24:18] spagewmf: and expect the system to "transform" your format? [21:25:43] nuria__: no. I see server/eventlogging/jrm.py it uses format to map just utc-millisec and uuid5-hex, so it's not general-purpose, hence I assume why it's not documented. [21:26:00] spagewmf: right [21:26:47] spagewmf: ok, i understand what you meant now. [21:32:17] (PS1) EBernhardson: Initial graphs of FlowReplies EventLogging schema [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/178646 [21:34:32] (PS2) EBernhardson: Initial graphs of FlowReplies EventLogging schema [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/178646 [21:35:48] mforns: lunching, will fix on mediawiki storage when i came back [21:36:04] np at all, [21:36:16] if you want, I can fix it too [21:43:14] ottomata: -- ping [21:47:40] piIIIIng [21:49:25] tnegrin: hiya [21:49:32] ohai [21:49:48] we were discussing about the moves to the analytics vlan [21:49:57] yesh [21:50:05] I was going to invite you to the hangout to discuss -- good time? [21:50:27] sure [21:51:00] we are checking to see if we have time for you [21:51:06] ;p [21:51:12] oh [21:51:13] ok [21:51:36] we'll contact you offline -- there are a few questions [21:51:45] sorry [21:51:49] I know you like meetings [21:51:55] ok [22:08:14] (PS3) EBernhardson: Initial graphs of FlowReplies EventLogging schema [analytics/limn-flow-data] - https://gerrit.wikimedia.org/r/178646 [22:30:31] Analytics-Refinery: Raw upload webrequest partitions for 2014-12-08T17/1H not marked successful - https://phabricator.wikimedia.org/T78081 (QChris) NEW p:Normal [22:31:13] Analytics-Refinery: Raw upload webrequest partitions for 2014-12-08T17/1H not marked successful - https://phabricator.wikimedia.org/T78081#835164 (QChris) [22:32:50] Analytics-Refinery: Raw upload webrequest partitions for 2014-12-08T17/1H not marked successful - https://phabricator.wikimedia.org/T78081#835177 (QChris) 2014-12-08T15/1H is tracked in T77999 2014-12-08T18/1H is tracked in T78000 [22:37:25] Analytics-Refinery: Raw upload webrequest partitions for 2014-12-08T17/1H not marked successful - https://phabricator.wikimedia.org/T78081#835197 (QChris) The partition is affected only during the 4 seconds from 2014-12-08T17:17:19 to 2014-12-08T17:17:22. This period matches kafka leader re-election to bring... [22:38:27] !log Marked raw upload webrequest partition for 2014-12-08T17/1H ok (See {{PhabT|78081}}) [22:52:33] (PS1) Ori.livneh: Initial commit. [analytics/statsv] - https://gerrit.wikimedia.org/r/178693 [22:52:43] (CR) Ori.livneh: [C: 2 V: 2] Initial commit. [analytics/statsv] - https://gerrit.wikimedia.org/r/178693 (owner: Ori.livneh) [22:54:45] (PS1) Ori.livneh: Initial commit of Python prototype [analytics/statsv] - https://gerrit.wikimedia.org/r/178694 [22:55:48] (CR) Ori.livneh: [C: 2 V: 2] Initial commit of Python prototype [analytics/statsv] - https://gerrit.wikimedia.org/r/178694 (owner: Ori.livneh) [23:03:07] Analytics-Refinery: Raw bits webrequest partition for 2014-12-09T17/1H not marked successful - https://phabricator.wikimedia.org/T78087 (QChris) NEW p:Normal [23:03:18] Analytics-Refinery: Raw webrequest partitions that were not marked successful due to configuration updates - https://phabricator.wikimedia.org/T74300#835282 (QChris) [23:03:19] Analytics-Refinery: Raw bits webrequest partition for 2014-12-09T17/1H not marked successful - https://phabricator.wikimedia.org/T78087#835271 (QChris) [23:04:24] !log Marked raw bits webrequest partition for 2014-12-09T17/1H ok (See {{PhabT|78087}}) [23:05:47] Analytics-Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://phabricator.wikimedia.org/T71615#835283 (QChris) Happened again for: 2014-12-09T12/2H (on text) [23:07:11] !log Marked raw text webrequest partitions for 2014-12-09T12/2H ok (See {{PhabT|71615|835283}}) [23:09:40] Analytics-Refinery: cp3009.esams.wikimedia.org lost a kafka message on 2014-12-09T14:33:05 - https://phabricator.wikimedia.org/T78088 (QChris) NEW p:Triage [23:10:22] !log Marked raw upload webrequest partition for 2014-12-09T14/1H ok (See {{PhabT|78088}}) [23:14:24] (PS1) Merlijn van Deen: CORS all /meta/ queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/178702 [23:20:30] nuria__: can we mark this as resolved: https://phabricator.wikimedia.org/T75030 [23:20:44] Analytics-Dashiki: Dashiki needs to have a friendlier mobile view. - https://phabricator.wikimedia.org/T75030#835331 (kevinator) p:Triage>Normal [23:26:44] Analytics-Dashiki: Dashiki needs to have a friendlier mobile view. - https://phabricator.wikimedia.org/T75030#835333 (Nuria) Open>Resolved [23:26:52] kevinator: yes, just did [23:27:03] nuria__: thanks [23:40:05] Analytics-Dashiki: Dashiki needs to have a friendlier mobile view. - https://phabricator.wikimedia.org/T75030#835387 (kevinator) [23:42:13] Analytics-Dashiki: weekly/monthly granularity in Dashiki - https://phabricator.wikimedia.org/T76092#835404 (kevinator) p:Triage>Normal [23:43:57] Analytics-Dashiki: Improve Dashiki's HTML template - https://phabricator.wikimedia.org/T73983#835417 (kevinator) p:Triage>Normal [23:44:17] Analytics-Dashiki: Icon font 404ing on metrics-staging - https://phabricator.wikimedia.org/T76747#835421 (kevinator) p:Triage>Normal [23:44:50] Analytics-Dashiki: Removing lines updates URL hash, but editing URL hash or using back/forward buttons has no effect - https://phabricator.wikimedia.org/T76746#835428 (kevinator) p:Triage>Normal [23:45:17] Analytics-Dashiki: Strange rendering glitches when removing lines - https://phabricator.wikimedia.org/T76745#835442 (kevinator) p:Triage>Normal [23:46:03] Analytics-Dashiki: Story: dashiki filters outliers - https://phabricator.wikimedia.org/T75316#835451 (kevinator) p:Triage>Normal [23:48:31] Analytics-Refinery: Implement functionality to have things like page_id in the X-Analytics field - https://phabricator.wikimedia.org/T77416#835459 (kevinator) p:Triage>High [23:49:19] Analytics-Refinery: Getting Ananth started - https://phabricator.wikimedia.org/T77196#835460 (kevinator) p:Triage>Normal [23:49:38] Analytics-Refinery: collect UDF requirements - https://phabricator.wikimedia.org/T77222#835461 (kevinator) p:Triage>Normal [23:50:02] Analytics-Refinery: Ananth to merge this change: https://gerrit.wikimedia.org/r/#/c/169346/ - https://phabricator.wikimedia.org/T77203#835462 (kevinator) p:Triage>High [23:55:13] Analytics-Dashiki: Remove the "confusing" under reported data for Edits and Pages Created in Vital Signs - https://phabricator.wikimedia.org/T75617#835475 (kevinator) [23:55:27] Analytics-Dashiki, Analytics-Wikimetrics: Remove the "confusing" under reported data for Edits and Pages Created in Vital Signs - https://phabricator.wikimedia.org/T75617#765748 (kevinator) [23:55:55] Analytics-Dashiki, Analytics-Wikimetrics, Analytics-Engineering: Remove the "confusing" under reported data for Edits and Pages Created in Vital Signs - https://phabricator.wikimedia.org/T75617#765748 (kevinator) [23:57:55] Analytics-Engineering: Pageview data files for mobile breakdowns: absence of data should not be represented as 'zero' - https://phabricator.wikimedia.org/T78025#835479 (kevinator) This seems related to T76406 I'll leave this as a distinct task to validate that Dashiki can handle what is dumped by Refinery. [23:58:41] Analytics-Dashiki, Analytics-Engineering: Pageview data files for mobile breakdowns: absence of data should not be represented as 'zero' - https://phabricator.wikimedia.org/T78025#835483 (kevinator) p:Triage>Normal [23:59:23] Analytics-Refinery, Analytics-Engineering: PageView reports by hive-webstatscollector should return undefined values when data is not available - https://phabricator.wikimedia.org/T76406#799656 (kevinator)