[00:09:27] <wikibugs>	 Analytics-Engineering, Analytics-Cluster: automate deletion of old logs in stat1002:/a/squid/archive/edits - https://phabricator.wikimedia.org/T87246#1002843 (kevinator)
[00:15:33] <kevinator>	 milimetric: yt?
[00:16:14] <kevinator>	 nuria: do you know how to reboot pentaho?
[00:17:14] <nuria>	 kevinator: resurrect you mean, cause reboot is not going to do it but yeah
[00:17:33] <kevinator>	 ahh
[00:17:52] <kevinator>	 yeah, it still has a pulse (502 bad gateway message)
[00:18:22] <nuria>	 kevinator: let me take a look
[00:19:10] <kevinator>	 thanks
[00:37:14] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Vital Signs user clicks "Add Metric" then Daily Pageviews (New Definition) - https://phabricator.wikimedia.org/T86141#1002897 (kevinator)
[00:38:26] <nuria>	 kevinator: sorry but something is up with labs now, machine is real slow and unresponsive (This is unrelated to pentaho)
[00:39:18] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Vital Signs user clicks "Add Metric" then Daily Pageviews (New Definition) - https://phabricator.wikimedia.org/T86141#962128 (kevinator)
[00:39:54] <nuria>	 kevinator: something is up with labs machine is very slow and unresponsive
[00:40:12] <kevinator>	 nuria: ahh
[00:40:13] <nuria>	 kevinator: (nothing to do with pentaho per se)
[00:40:20] <kevinator>	 ok
[00:40:45] <nuria>	 kevinator: i cannot even use the command line so i guess this is going to have to wait
[00:41:12] <kevinator>	 Carolynne was wanting to use it.  I’ll let her know it will not get resolved tonight and we’ll see if labs is better tomorrow
[00:55:34] <nuria>	 ottomata: yt?
[00:58:51] <ottomata>	 yes but deep in some code!
[01:18:58] <nuria>	 ottomata: ok, lemme know when you are out of the deepness
[01:22:20] <ottomata>	 hm, nuria go ahead
[01:22:34] <nuria>	 ottomata: looked at data for mobile jobs
[01:22:55] <nuria>	 https://www.irccloud.com/pastebin/umVJWz8s
[01:23:31] <ottomata>	 ah ya?
[01:23:43] <ottomata>	 hm,
[01:23:44] <ottomata>	 the 0 pad didn't come out, eh?
[01:23:49] <nuria>	 and the only file that looks right is 2015-01-28.gz and i am wondering what is up with "/wmf/data/archive/mobile_apps/uniques/daily/2015-1-29.gz" cause that looks is from an "older" job
[01:23:56] <nuria>	 no, it did for the 28th
[01:24:18] <nuria>	 the 28th has "good" data
[01:24:29] <nuria>	 ottomata: rest of files look from prior runs
[01:25:04] <ottomata>	 hm, btw, nuria, do you have a bunch of test coordinators left in running state for this?
[01:25:22] <nuria>	 ottomata: no, i should have kill them all
[01:25:35] <ottomata>	 $ oozie jobs  -jobtype coordinator -filter status=RUNNING | grep mobile_apps_daily_uniques-coord
[01:25:35] <ottomata>	 0016981-150116220452628-oozie-oozi-C     mobile_apps_daily_uniques-coordRUNNING   1    DAY          2015-01-28 00:00 GMT    2015-01-31 00:00 GMT
[01:25:35] <ottomata>	 0004041-150116220452628-oozie-oozi-C     mobile_apps_daily_uniques-coordRUNNING   60   MINUTE       2015-01-08 00:00 GMT    2015-01-30 02:00 GMT
[01:25:35] <ottomata>	 0004028-150116220452628-oozie-oozi-C     mobile_apps_daily_uniques-coordRUNNING   60   MINUTE       2015-01-08 00:00 GMT    2015-01-30 02:00 GMT
[01:25:35] <ottomata>	 0003988-150116220452628-oozie-oozi-C     mobile_apps_daily_uniques-coordRUNNING   60   MINUTE       2015-01-08 00:00 GMT    2015-01-30 02:00 GMT
[01:25:35] <ottomata>	 0000265-150116220452628-oozie-oozi-C     mobile_apps_daily_uniques-coordRUNNING   60   MINUTE       2015-01-08 00:00 GMT    2015-01-30 02:00 GMT
[01:25:35] <ottomata>	 0000015-150116163703781-oozie-oozi-C     mobile_apps_daily_uniques-coordRUNNING   60   MINUTE       2015-01-08 00:00 GMT    2015-01-30 02:00 GMT
[01:25:36] <ottomata>	 0026318-141210154539499-oozie-oozi-C     mobile_apps_daily_uniques-coordRUNNING   1    DAY          2015-01-16 00:00 GMT    2015-01-31 00:00 GMT
[01:25:39] <nuria>	 I kill them with " oozie jobs  -jobtype coord | grep mobile |awk '{print $1}' | xargs oozie job -kill"
[01:26:41] <nuria>	 ok, not sure how those survived but they shoudl be killed and we should have just 1 running
[01:26:54] <nuria>	 if I test i will prefix them with "TESTING"
[01:27:14] <ottomata>	 yeah, hm, i think maybe i didn'g kill one too?
[01:27:23] <ottomata>	 i see a daily one running since 01-15
[01:27:27] <ottomata>	 just killed that
[01:27:36] <ottomata>	 that looks like one I submitted?
[01:27:37] <ottomata>	 hm
[01:28:08] <ottomata>	 I'm going to kill them all, and restart this job, and remove the 01-28 data and start it from there
[01:28:10] <ottomata>	 see how it does
[01:28:13] <ottomata>	 s'ok?
[01:28:15] <nuria>	 ottomata: I do not see those doing oozie jobs  -jobtype coord | grep mobile
[01:28:46] <nuria>	 ottomata: ok
[01:28:46] <ottomata>	 i just killed them
[01:28:46] <ottomata>	 i think
[01:29:15] <nuria>	 ottomata: ok, then data as of tomorrow should be good
[01:29:24] <ottomata>	 i'll start it for yesterday
[01:29:27] <ottomata>	 so it launches immediately
[01:29:57] <nuria>	 ottomata: sounds great
[01:30:17] <ottomata>	 nuria
[01:30:17] <ottomata>	 0023327-150116220452628-oozie-oozi-C
[01:30:24] <ottomata>	 fingers crossed
[01:30:43] <nuria>	 ottomata: sorry about the old jobs but I thought all displayed whne doing: oozie jobs  -jobtype coord | grep mobile
[01:30:48] <ottomata>	 oh cool, it launched yesterday and today
[01:30:55] <nuria>	 ottomata: of course!
[01:30:58] <ottomata>	 i would think that would work, nuria
[01:30:58] <ottomata>	 dunno
[01:31:34] <nuria>	 ottomata: thank you. will look at data in a bit and see how it looks
[01:39:32] <ottomata>	 k cool
[02:29:59] <fhocutt>	 it always disturbs me when I appear to have fixed a bug but don't understand why.
[06:10:45] <grrrit-wm>	 (PS10) Ananthrk: Added Geocode core class that uses Maxmind-V2 Added GeocodedCountryUDF to get country code from IP address Added GeocodedDataUDF to get Geo data from IP address Added new dependency maxmind-geoip2 in refinery-core Added Maxmind's test databases to resourc [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551
[06:39:52] <grrrit-wm>	 (PS1) Ananthrk: Added IpUtil class for IP address related utility methods Added ClientIpUDF to get client IP address from source IP and X-Forwarded-For header values [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187651
[08:15:16] <wikibugs>	 Analytics-Engineering: Add time range selection to Limn dashboards (or new Dashiki dashboards) - https://phabricator.wikimedia.org/T87603#1003892 (Tgr) This would be useful to double-check assumptions, amongst other things. Currently creating a graph is expensive so one typically makes one graph for a given i...
[15:25:43] <wikibugs>	 Analytics-Engineering, Analytics-Wikimetrics: Wikimetrics should run latest puppet w/o local changes - https://phabricator.wikimedia.org/T88116#1004281 (kevinator) p:Triage>High
[15:26:52] <wikibugs>	 Analytics-Engineering, Analytics-Wikimetrics: Wikimetrics should run latest puppet w/o local changes - https://phabricator.wikimedia.org/T88116#1004284 (Nuria) Remember: git pull --rebase when pulling
[15:46:29] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Vital Signs user clicks "Add Metric" then Daily Pageviews (New Definition) - https://phabricator.wikimedia.org/T86141#1004318 (Nuria) There are 4 different steps:  #1 Pageview data goes into refined tables (done) #2 oozie job to compute pageview agr...
[15:49:38] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Vital Signs user clicks "Add Metric" then Daily Pageviews (New Definition) - https://phabricator.wikimedia.org/T86141#1004322 (Nuria) Some oozie info: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie Code for other oozie jobs: https://git...
[15:52:05] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Vital Signs user clicks "Add Metric" then Daily Pageviews (New Definition) - https://phabricator.wikimedia.org/T86141#1004324 (Nuria) More docs: https://wikitech.wikimedia.org/wiki/Analytics/Pagecounts-all-sites  Code for step #3 (python) https://gi...
[16:39:24] <wikibugs>	 Wikidata, Analytics: active user statistics that have less lag than wikistats - https://phabricator.wikimedia.org/T88121#1004364 (JanZerebecki) NEW
[17:36:50] <mforns>	 nuria, pulling with --rebase is only for production? or is it also for staging?
[17:36:59] <nuria>	 mforns: both
[17:37:11] <nuria>	 mforns: staging connects to prod enwiki just teh same
[17:37:21] <nuria>	 it's just that the wikimetrics db is different
[17:37:26] <mforns>	 aha
[17:38:11] <mforns>	 so I pulled with --rebase in staging, but I can not see any file with changes...
[17:38:24] <mforns>	 what files should be different from origin production, nuria ?
[17:38:52] <nuria>	 mforns: there should be a commit on top with passwords i believe,
[17:38:58] <nuria>	 so you cann connect to db
[17:39:06] <nuria>	 as passwords are not on puppet
[17:40:20] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Analytics Engineer has an oozie job to aggreate page views by time - https://phabricator.wikimedia.org/T88125#1004447 (kevinator) NEW
[17:40:27] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Analytics Engineer has an oozie job to aggreate page views by time - https://phabricator.wikimedia.org/T88125#1004447 (kevinator) Some oozie info: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie
[17:41:27] <mforns>	 nuria, thanks, anyway I don't see the files, I'll have a look at production and see if I can find them
[17:41:47] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Analytics Engineer has a python job to aggregate by project - https://phabricator.wikimedia.org/T88127#1004465 (kevinator) NEW
[17:44:24] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Vital Signs user has access to the new page view data  - https://phabricator.wikimedia.org/T88128#1004474 (kevinator) NEW
[17:44:49] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Vital Signs user has access to the new page view data - https://phabricator.wikimedia.org/T88128#1004474 (kevinator)
[17:47:26] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Analytics Engineer has a python job to aggregate by project - https://phabricator.wikimedia.org/T88127#1004484 (kevinator)
[17:54:26] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Vital Signs user clicks "Add Metric" then Daily Pageviews (New Definition) - https://phabricator.wikimedia.org/T86141#1004505 (kevinator)
[17:54:32] <mforns>	 nuria, found them
[17:55:23] <nuria>	 mforns: ok
[17:59:34] <DarTar>	 hey mforns
[17:59:42] <mforns>	 hi DarTar
[18:00:01] <DarTar>	 safely back in paradise island?
[18:00:07] <mforns>	 xD, yes
[18:00:13] <DarTar>	 good
[18:00:17] <mforns>	 a bit jet-lagged [again]
[18:00:33] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Analytics Engineer has an oozie job to aggreate page views by time - https://phabricator.wikimedia.org/T88125#1004517 (kevinator)
[18:01:36] <DarTar>	 quick one:
[18:01:51] <DarTar>	 I noticed that the latest cube (v.0.4) in Pentaho has garbled country names for non ASCII characters
[18:04:03] <DarTar>	 mforns: ^
[18:04:22] <mforns>	 sorry, DarTar
[18:04:37] <mforns>	 I see
[18:05:16] <mforns>	 havin a look DarTar
[18:05:33] <DarTar>	 cool thx
[18:17:18] <qchris>	 ottomata: The cron that cleans Hive data will kill the data for 2015-01-01, but I could not yet backfill the glam_nara tsvs, as the geocoding udf is not yet merged.
[18:17:35] <qchris>	 Do you think it'd be ok if I switched the 31 deletion interval to 35 days?
[18:18:03] <qchris>	 (That way the udf could get more time to review/merge, and I could still backfill afterwards)
[18:18:20] <qchris>	 Meh ... s/35/40/
[18:21:43] <ottomata>	 qchris, i think that's fine?  or, i am ready to merge that thing today i think
[18:21:46] <ottomata>	 sorry, just talking to toby right now
[18:21:55] <ottomata>	 we don't have the IP udf yet though
[18:22:07] <qchris>	 The glam_nara does not use the IP stuff. So that's ok
[18:22:27] <qchris>	 (I know that this is a broken approach ... but that's what glam files currently do)
[18:22:52] <qchris>	 Happy chatting with tneg-rin. I'll upload something to puppet then for review.
[18:22:53] <qchris>	 Thanks
[18:23:19] <wikibugs>	 MediaWiki-ContentHandler, Analytics-EventLogging: [Regression 1.25wmf14] HTML of JSON blocks (e.g. Schema pages) appears garbled - https://phabricator.wikimedia.org/T86748#1004574 (Jdforrester-WMF)
[18:24:48] <grrrit-wm>	 (PS1) Gilles: Add missing post-merge script [analytics/multimedia] - https://gerrit.wikimedia.org/r/187712
[18:25:06] <grrrit-wm>	 (CR) Gilles: [C: 2 V: 2] Add missing post-merge script [analytics/multimedia] - https://gerrit.wikimedia.org/r/187712 (owner: Gilles)
[18:31:02] <grrrit-wm>	 (PS1) Gilles: Fix lack of metric wildcard [analytics/multimedia] - https://gerrit.wikimedia.org/r/187715
[18:31:18] <grrrit-wm>	 (CR) Gilles: [C: 2 V: 2] Fix lack of metric wildcard [analytics/multimedia] - https://gerrit.wikimedia.org/r/187715 (owner: Gilles)
[18:31:23] <grrrit-wm>	 (Merged) jenkins-bot: Fix lack of metric wildcard [analytics/multimedia] - https://gerrit.wikimedia.org/r/187715 (owner: Gilles)
[18:33:17] <milimetric>	 nuria: in terms of viz, I was kind of leaning towards sunburst: http://bl.ocks.org/kerryrodden/7090426
[18:33:27] <nuria>	 lemmee seeee
[18:33:40] <nuria>	 NICE!!!!
[18:33:40] <milimetric>	 with filters above, where people could filter down dates, wikis, etc.
[18:33:52] <nuria>	 i am such a sucker for this kind of thing!!!
[18:34:10] <milimetric>	 :) I like to find the perfect graph, so I'm still a little torn
[18:34:33] <nuria>	 Iteration man, when they see this they will LOVE you
[18:39:53] <wikibugs>	 Analytics-Dashiki, Analytics-Engineering, Analytics-Cluster: Analytics Engineer has a python job to aggregate by project - https://phabricator.wikimedia.org/T88127#1004629 (Ottomata) Does this have to be a "Python job"?  Maybe we should remove that as a requirement.
[18:45:13] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Added Geocode core class that uses Maxmind-V2 Added GeocodedCountryUDF to get country code from IP address Added GeocodedDataUDF to get Geo [analytics/refinery/source] - https://gerrit.wikimedia.org/r/183551 (owner: Ananthrk)
[18:45:17] <ottomata>	 merged!
[18:45:20] <ottomata>	 qchris: % :D
[18:45:21] <ottomata>	 ^
[18:45:26] <qchris>	 \o/
[18:45:28] <qchris>	 Thanks.
[18:49:43] <milimetric>	 nuria: thinking about the 1000 minimum bucket size rule of thumb
[18:49:52] <milimetric>	 so in the case of a funnel
[18:50:05] <milimetric>	 do you think the first step of the funnel should have at least 1000?  Or the last step?
[18:50:08] <milimetric>	 I was thinking first step
[18:51:50] <kevinator>	 milimetric: I think the last step… especially if the drop off from 1st step to 2nd step is huge (10% make it to the second step)
[18:52:12] <milimetric>	 kevinator: then we would be discounting the vast majority of data
[18:52:24] <milimetric>	 because it's mostly granular and the drop-off is pretty big
[18:52:28] <milimetric>	 or is that not what you meant
[18:54:15] <kevinator>	 I’m not sure what you mean by “discounting"
[18:58:23] <milimetric>	 kevinator: sorry, was making sure I can join the VE QR
[18:58:40] <milimetric>	 kevinator: ok, so if we require 1000 hits or more to the last step of the funnel
[18:59:08] <milimetric>	 then that means we have to not count, or discount the majority of wikis
[18:59:30] <milimetric>	 all besides frwiki, itwiki, and two others I forget
[18:59:49] <milimetric>	 and on some days, even those don't have enough data
[19:00:20] <kevinator>	 ahhh, ok
[19:01:54] <mforns>	 DarTar, the country names are fixed
[19:02:39] <kevinator>	 milimetric: are the events sampled?
[19:03:33] <DarTar>	 mforns: awesome, thanks
[19:03:55] <mforns>	 DarTar, yw!
[19:06:56] <nuria>	 milimetric: sorry i missed the ping
[19:07:11] <nuria>	 milimetric: what are we going to calculate with teh funnel data
[19:07:24] <nuria>	 <count on step 1> /<count of step 2>
[19:07:31] <nuria>	 a rate of that sort?
[19:07:36] <milimetric>	 kevinator: not sure at what rate
[19:07:44] <milimetric>	 nuria: yes, for now
[19:08:44] <nuria>	 milimetric: the "bucket" rules i have used when calculating means and percentiles, in the case of rates we just have to decide what is a statistically significant dataset
[19:09:30] <milimetric>	 i guess for now I can just display the absolute counts along with the rates, and we can implement a "filter out lower than X" feature
[19:10:04] <nuria>	 milimetric: but we should not show a wiki with 100 points, that data is meaningless
[19:10:27] <milimetric>	 ok, so i'll make that a minimum for now
[19:11:11] <grrrit-wm>	 (PS1) QChris: Bump version to 0.0.3 to include Geocoding UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187721
[19:11:12] <grrrit-wm>	 (PS1) QChris: Bump working version to 0.0.5-SNAPSHOT, now that 0.0.4 is released [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187722
[19:12:14] <nuria>	 leila: yt?
[19:12:21] <grrrit-wm>	 (CR) QChris: [C: -2] Bump version to 0.0.3 to include Geocoding UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187721 (owner: QChris)
[19:12:36] <leila>	 yup, wanna talk nuria?
[19:12:42] <ottomata>	 qchris: -2?  also, commit message wrong version?
[19:12:45] <nuria>	 leila; one side question
[19:12:52] <grrrit-wm>	 (PS2) QChris: Bump version to 0.0.4 to include Geocoding UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187721
[19:12:54] <grrrit-wm>	 (PS2) QChris: Bump working version to 0.0.5-SNAPSHOT, now that 0.0.4 is released [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187722
[19:13:00] <qchris>	 ottomata:  Yes :-)
[19:13:08] <leila>	 nuria: sure.
[19:13:14] <nuria>	 for milimetrics work with VE with similar to yours with wikigrok ^leila
[19:13:17] <grrrit-wm>	 (CR) QChris: Bump version to 0.0.4 to include Geocoding UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187721 (owner: QChris)
[19:13:25] <leila>	 yes, I realized that nuria.
[19:13:25] <qchris>	 ottomata: now it should be good.
[19:13:52] <nuria>	 leila, milimetric : we are calculating rates like "number of users on edit step"/"number of users on save step"
[19:13:56] <ottomata>	 qchris: there is also now a Webrequest class and corresponding UDFs
[19:13:58] <leila>	 I wasn't sure if VE's problem was specific to them but milimteric's email suggested similar anomalies in their data.
[19:14:04] <ottomata>	 that do things like extract xanalytics values, etc.
[19:14:07] <ottomata>	 add that to changelog?
[19:14:23] <qchris>	 I did add that. Didn't I?
[19:14:26] <nuria>	 leila: in order to extrapolate a raye seems to me we need a minimum statistically significant dataset
[19:14:37] <ottomata>	 oOOp
[19:14:37] <nuria>	 leila, milimetric : a rate
[19:14:38] <ottomata>	 you did
[19:14:40] <qchris>	 "Also UDFs to get webrequest's access method, extract values from the X-Analytics header and determine whether or not the request came from a crawler..."
[19:14:44] <ottomata>	 sorry, it was a long sentence :p
[19:14:48] <qchris>	 Yes :-(
[19:14:55] <qchris>	 Feel free to rephrase.
[19:14:56] <leila>	 nuria: the data-set should be large enough for that kind of analysis, right?
[19:15:01] <ottomata>	 tha'ts fine, i would have added multiple items for that
[19:15:01] <milimetric>	 right
[19:15:04] <ottomata>	 but whatever, i don't really care
[19:15:05] <ottomata>	 cool!
[19:15:14] <nuria>	 leila, milimetric : i normally have used buckets sizes yo calculate means / percentiles and such
[19:15:29] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Bump version to 0.0.4 to include Geocoding UDFs [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187721 (owner: QChris)
[19:15:36] <qchris>	 Oh ... I thought ... meh. it seems I mis-understood your changelog format then.
[19:15:36] <nuria>	 leila, milimetric : something like: for a percentile 90 you need at least 1000 samples
[19:15:53] <nuria>	 leila: should we apply similar rules here?
[19:15:53] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Bump working version to 0.0.5-SNAPSHOT, now that 0.0.4 is released [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187722 (owner: QChris)
[19:15:57] <leila>	 nuria: are you worried that you don't have enough data?
[19:16:03] <ottomata>	 qchris: there is no particular format
[19:16:06] <leila>	 cuz otherwise, you can use all the data we have, nuria
[19:16:25] <qchris>	 :-D
[19:16:30] <nuria>	 leila, milimetric : i am worried there is very little data for small wikis for a daily value
[19:16:33] <ottomata>	 qchris: you gonna tag, etc?  you just reminded me to make a phab task for something...
[19:16:54] <qchris>	 Yes, I'll tag. But could you take care of uploading to archive? (I have no account there)
[19:17:00] <leila>	 nuria: wikigrok was only on in enwiki so we're fine for that one with using the whole data.
[19:17:20] <leila>	 Editing team's data is different since they have it on everywhere.
[19:17:23] <nuria>	 leila, milimetric: but for VE for some wikis there is less than a 1000 points per day right milimetric ?
[19:17:38] <qchris>	 (tagging done)
[19:17:53] <wikibugs>	 Analytics-Engineering, Analytics-Cluster: Streamline analytics/refinery/source release and deployment process - https://phabricator.wikimedia.org/T88137#1004755 (Ottomata) NEW
[19:17:57] <milimetric>	 nuria, leila: yes, some wikis only have like 2 points per day
[19:17:57] <ottomata>	 qchris: yes
[19:18:01] <qchris>	 Thanks.
[19:18:12] <ottomata>	 if I can login....its been hanging on that recently
[19:18:42] <qchris>	 Why cannot Java Webservers behave and be responsive? :-/
[19:19:14] <leila>	 nuria: in this case, with 2 points you can't do much. I suggest that you divide the effort in two parts: 1) for wikis with many data points, use your gestimate approach, 1000 data points provide n% confidence. For smaller wikis, either ignore them for now, or see how much confidence you can get from the data we have and report results with confidence attached to them.
[19:19:21] <chasemp>	 qchris: http://en.wikipedia.org/wiki/The_Scorpion_and_the_Frog :)
[19:20:02] <nuria>	 leila, milimetric: ok, then let's go with bucket sizes than allow us a higher confidence, let's start at a 1000
[19:20:37] <milimetric>	 leila, nuria: this seems a bit arbitrary to me now... why not 100?
[19:21:12] <nuria>	 milimetric: because a 1000 interval gives you a confidence to calculate a percentile90 over your time-based dataset of rates
[19:21:18] <leila>	 because 100 will give you a very low confidence, milimetric.
[19:21:40] <nuria>	 milimetric, leila: There are other strategies , for example" If more than 20 percent (or one out of five) of the test-execution results appear not to be similar to the others, something is generally wrong with the test environment, the application, or the test itself."
[19:22:12] <nuria>	 milimetric, leila: this is also a rule of thumb i used before
[19:22:49] <leila>	 yeah, nuria. nuria, milimetric, let's start with wikis that have at least 1000 data points. If figuring out the issue based on those doesn't explain the issue for smaller wikis, then we will revisit to dig more?
[19:23:00] <milimetric>	 leila: ok, so back to my initial question :)
[19:23:03] <nuria>	 milimetric, leila: much agreed.
[19:23:13] <milimetric>	 if we're computing a simple funnel, A -> B
[19:23:25] <milimetric>	 and there are 1200 people that get to A
[19:23:29] <milimetric>	 and 900 people that get to B
[19:23:33] <milimetric>	 do we use that bucket?
[19:24:01] <nuria>	 milimetric, leila: i would say we need at least 1000 points on "final" step to be able to "infer" anything
[19:24:18] <leila>	 the answer to this depends on which part of the funnel you're suspicious about, milimetric.
[19:24:41] <milimetric>	 leila: we're computing rate of A->B, so we need % got to B
[19:24:52] <leila>	 If you have no idea, I'd go with nuria's suggestion of requiring 1000 points on the final step
[19:25:01] <nuria>	 leila: it's a two step funnel for now
[19:25:07] <nuria>	 leila: so simple
[19:25:10] <milimetric>	 it's not
[19:25:15] <milimetric>	 i'm simplifying
[19:25:17] <nuria>	 sorry, simpler
[19:25:26] <leila>	 can we go to the batcave, nuria, milimetric?
[19:25:29] <nuria>	 sure
[19:25:33] <milimetric>	 i'm sadly in a different meeting
[19:25:41] <nuria>	 ok, later then
[19:25:45] <leila>	 ooki. should we talk after your meeting milimetric?
[19:25:51] <nuria>	 you ping us
[19:26:53] <milimetric>	 ok, it'll be a while, I'll ping
[19:27:06] <nuria>	 leila: regarding wikigrok
[19:27:16] <leila>	 yes, yes, nuria.
[19:27:39] <nuria>	 the # of events with "widget-impression" w/o "page-impression" for say one day (just picked the 19th of Jan)
[19:27:44] <nuria>	 it's pretty large
[19:27:55] <leila>	 yes!
[19:27:58] <leila>	 yes, nuria.
[19:28:19] <nuria>	 of " 886720" widget-impression only 320222 have a "page-impression"
[19:28:26] <leila>	 I'm also digging to see if there is something wrong with taskToken.
[19:28:41] <nuria>	 i tested that there is no event drops and there is not, what is on teh db is on the logs
[19:28:48] <leila>	 hokay, that's huge, nuria. :-|
[19:29:18] <nuria>	 i am going to test couple more things and will let you know but I think their tokens are not doing what they would expect
[19:29:18] <leila>	 I see, nuria.
[19:30:08] <nuria>	 cause the "volume" makes sense. see totals (for 1 day) : "page-impression": 1612105 "widget-impression": 886720
[19:30:22] <nuria>	 so "twice as many people on step #1 than step#2"
[19:30:28] <nuria>	 ^leila
[19:30:38] <nuria>	 but after what doesn't match is the sequence
[19:31:06] <nuria>	 i am going to do another match (base on something that is not the user token) and will let you know
[19:31:29] <leila>	 yes, the volume is not alarming.
[19:32:15] <nuria>	 but tokens are wrong, we should tell kaldari when he gets in
[19:32:29] <leila>	 nuria, if you group_concat(event_action) and group by event_userToken, you see a lot of events with more than one type of action, mostly page-impression, in them.
[19:32:47] <leila>	 that's a problem, nuria, and it's most probably what you say. the userToken doesn't work properly.
[19:33:17] <leila>	 I'll ping him to loop in
[19:33:38] <nuria>	 leila: will spend another hour on this and let you know
[19:33:45] <leila>	 okay. thanks!
[19:37:00] <ottomata>	 !log released refinery 0.0.4
[19:39:39] <grrrit-wm>	 (PS1) Ottomata: Update refinery artifacts to 0.0.4 [analytics/refinery] - https://gerrit.wikimedia.org/r/187729
[19:40:10] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Update refinery artifacts to 0.0.4 [analytics/refinery] - https://gerrit.wikimedia.org/r/187729 (owner: Ottomata)
[20:21:38] <ottomata>	 !log deployed refinery 0.0.4
[20:21:42] <ottomata>	 hm, i guess logbot is not listening
[20:21:45] <ottomata>	 analytics-logbot: hello?
[20:21:45] <analytics-logbot>	 I am a logbot running on tools-exec-01.
[20:21:45] <analytics-logbot>	 Messages are logged to www.mediawiki.org/wiki/Analytics/Server_Admin_Log.
[20:21:45] <analytics-logbot>	 To log a message, type !log <msg>.
[20:21:55] <ottomata>	 oh it is
[20:21:58] <ottomata>	 just no ack.
[20:21:58] <ottomata>	 cool
[21:00:14] <nuria>	 leila: yt?
[21:00:22] <leila>	 yes, nuria
[21:01:09] <nuria>	 ok, so i do not have much to add to what i said before if you look at events by userToken they do not match
[21:01:48] <nuria>	 i looked at the code for the token and it is clearly labeled with 'do not trust this to be unique'
[21:01:53] <leila>	 taskToken or userToken, nuria?
[21:02:00] <nuria>	 userToken
[21:02:14] <leila>	 doh! then the sampling will also be problematic, nuria, right?
[21:02:36] <leila>	 batcave nuria?
[21:02:41] <leila>	 (with a link)
[21:02:56] <nuria>	 if it is purely random like if i< 3 and i  = rand(0,3) nop
[21:03:10] <nuria>	 give me 5 mins
[21:03:13] <leila>	 sure
[21:03:17] <leila>	 me, too
[21:06:09] <nuria>	 leila, batcave whenever you are ready: https://plus.google.com/hangouts/_/wikimedia.org/a-batcave
[21:15:36] <kevinator>	 mforns: are you working on this task: T88116
[21:18:47] <leila>	 milimetric: we're in batcave to discuss sampling
[21:24:56] <mforns>	 kevinator, yes
[21:25:30] <kevinator>	 mforns: do you mind assigning the bug to yourself?  It helps when me when I’m triaging bugs :-)
[21:27:01] <mforns>	 kevinator, sure
[21:27:20] <wikibugs>	 Analytics-Engineering, Analytics-Wikimetrics: Wikimetrics should run latest puppet w/o local changes - https://phabricator.wikimedia.org/T88116#1005106 (mforns) a:mforns
[21:30:33] <nuria>	 leila, found the bug about randomness: https://phabricator.wikimedia.org/T78449
[21:30:45] <leila>	 thanks nuria
[21:31:48] <nuria>	 leila: also, let me re-state, what is on the logs is on the db so there are no validation issues or dropped event issues
[21:32:13] <leila>	 yes, nuria. thanks! :-)
[21:32:54] <nuria>	 leila: also tasktoken is also generated from this method: taskToken: mw.user.generateRandomSessionId(),
[21:33:04] <nuria>	 so both will suffer teh same issues
[21:33:34] <leila>	 I see. nuria, do you know who can work on this issue?
[21:33:44] <milimetric>	 leila, nuria, coming
[21:34:28] <milimetric>	 oh, you're not there anymore :)
[21:35:38] <leila>	 nuria: can you come back to batcave, kevinator, join as well, if you're interested
[21:36:38] <nuria>	 leila: no, not really, kaldari would know, but note that this might account for lack of uniqueness of ad most say 10/100 (i found 1/1000 but being super conservative here cause also the mobile browser base it's more skewed towards safari, it's probably less than 10/100)
[21:36:52] <nuria>	 leila: let me estimate that
[21:38:34] <milimetric>	 nuria: wanna jump in the hangout, we're here
[21:38:40] <nuria>	 k
[22:07:14] <nuria>	 leila: so the lack of randomness is actually more important in mobile as a LOT of traffic in enwiki comes from safari, so actually it is a bigger problem in mobile (wikigrok) than desktop
[22:07:30] <nuria>	 leila: need to eat, will try to ctach kaldari later
[22:08:11] <nuria>	 ottomata: holaaa, can we delete the old data for the mobile jobs?
[22:08:34] <ottomata>	 oh yes
[22:08:42] <ottomata>	 want me to re run some of them?
[22:08:45] <ottomata>	 i can launch a backfill job
[22:09:09] <nuria>	 ottomata: that will be awesome, 28th and 29th look good
[22:09:17] <nuria>	 everything else can be scrapped
[22:09:36] <nuria>	 ottomata: how do i do so those files are available through the mount in 1002?
[22:09:57] <ottomata>	 ls /mnt/hdfs/wmf/data/archive/mobile_apps/uniques/daily
[22:10:25] <nuria>	 ooohhhhh MAGIC!
[22:10:38] <nuria>	 ottomata: ok, will check new jobs when i came back from lunch
[22:10:44] <nuria>	 ottomata: thank you!
[22:12:20] <ottomata>	 nuria: oozie job -info 0025430-150116220452628-oozie-oozi-C
[22:15:49] <leila>	 nuria: I'll go eat, too, and then dig a bit in theory. will touch-base later in the day
[22:15:52] <leila>	 ciao.
[22:18:33] <grrrit-wm>	 (PS1) Ottomata: Add use of versions-maven-plugin and maven-scm-plugin [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187820 (https://phabricator.wikimedia.org/T88137)
[22:44:34] <grrrit-wm>	 (PS2) Ottomata: Add use of versions-maven-plugin and maven-scm-plugin [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187820 (https://phabricator.wikimedia.org/T88137)
[22:45:42] <ottomata>	 woops
[22:46:10] <ottomata>	 did not mean to do that!
[22:50:16] <grrrit-wm>	 (PS1) Ottomata: Oops, reverting to 0.0.5-SNAPSHOT [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187826
[22:50:37] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Oops, reverting to 0.0.5-SNAPSHOT [analytics/refinery/source] - https://gerrit.wikimedia.org/r/187826 (owner: Ottomata)
[22:54:23] <grrrit-wm>	 (PS2) Jdlrobson: Update scripts in light of recent changes [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/181428