[00:46:40] (PS2) Mforns: [WIP] Add support for wiki explosion and others. [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/200239 [00:47:52] (CR) Mforns: "Second patch adds support for wiki explosion." [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/200239 (owner: Mforns) [06:14:34] (CR) KartikMistry: [C: 2] Added README with new language addition how-to [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/200165 (owner: KartikMistry) [06:14:41] (Merged) jenkins-bot: Added README with new language addition how-to [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/200165 (owner: KartikMistry) [10:07:05] Hi team ! [10:07:27] I'm finally back :) [14:00:41] joel , yt? [14:01:18] Yup, joining [14:01:31] Hi nuria [14:14:46] Analytics, Analytics-Cluster, Patch-For-Review: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1162520 (JAllemandou) a:Ottomata>JAllemandou [14:14:59] Analytics-Kanban, Analytics-Wikimetrics, Patch-For-Review: Get a measure of daily usage of wikimetrics by userbase - https://phabricator.wikimedia.org/T94193#1162521 (kevinator) p:Triage>Normal [14:18:31] Analytics, Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1162539 (JAllemandou) [14:23:49] (PS3) Joal: Add x_cache, ua_map and x_analytics_map fields to refine table. [analytics/refinery] - https://gerrit.wikimedia.org/r/198789 [14:24:30] (PS4) Joal: Add x_cache, ua_map and x_analytics_map fields to refine table. [analytics/refinery] - https://gerrit.wikimedia.org/r/198789 [14:24:37] Forgot to rebase ... sorry [14:47:07] joal: so xcache has data like "cp1067 miss (0), amssq38 hit (44), amssq62 frontend miss (0)" and we will be adding it "as is" to refined tables, correct? [14:47:32] Indeed nuria [14:47:42] I wondered if there was a better wasy to format [14:48:26] But from what Andrew said, maybe waiting for a new format provided by cachig system, then parse as map [14:51:10] (CR) Nuria: Add x_cache, ua_map and x_analytics_map fields to refine table. (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/198789 (owner: Joal) [14:53:15] nuria, joal, i'm reviewing that patch now too [14:53:32] nuria: re: your comment about x_analytics. he does that 3 lines below, no? [14:53:43] 6 lines [14:54:10] and, joal: can we call ua_map user_agent_map for consistency? [14:54:23] ottomata: Yes, for sure ! [14:54:31] Forgot about this onen [14:54:39] ottomata, joal: excuse my retardation [14:55:30] ottomata: for consistency again, ua_parser function should become user_agent_parser ? [14:56:00] otto: this jar: spark-shell --jars /home/otto/algebird-core_2.10-0.9.0.jar isn't going to need to be on archiva so mforns can run his code on the cluster? [14:58:48] oh, the tempoary function? [14:59:01] i don't really care about temporary functino name [14:59:12] ok, np then :) [14:59:14] but, i personally would keep that as ua_parser [14:59:19] just cause that is what the library is named [14:59:21] cool [14:59:22] UAParser [14:59:28] yes. [14:59:29] nuria: [14:59:36] Anything else before I push again ? [14:59:44] we need to add algebird to archiva and then add it is a dep in the refinery-jobs/ pom.xml [14:59:53] joal: looking [15:00:58] joal: just note in the changelog that you are adding refinery-*-0.0.9.jar s [15:01:08] sorry [15:01:09] not changelog [15:01:10] commit message [15:01:16] you say you are using them, but it would be nice if the commit message explicitly said it [15:01:23] Yes, will do [15:01:49] if qch ris were listening he would suggest to make all these things different commits (like 5 commits?!), but i am not too worried about that :) [15:02:24] ottomata: And he would be right .. [15:02:54] Since you don't mind, I'll keep it this way [15:04:09] yeah [15:04:11] i don't mind :) [15:04:20] i find it annoying sometimes to have to separate them all, especially with gerrit and code review [15:05:13] Yeah, cleaner and easier to rollback, but less cool to commit / review [15:05:45] (PS5) Joal: Add x_cache, ua_map and x_analytics_map fields to refine table. [analytics/refinery] - https://gerrit.wikimedia.org/r/198789 [15:07:41] (CR) Ottomata: [C: 2 V: 2] Add x_cache, ua_map and x_analytics_map fields to refine table. [analytics/refinery] - https://gerrit.wikimedia.org/r/198789 (owner: Joal) [15:07:49] thar ya go :) [15:07:58] Thx Andrew :) [15:08:16] nuria: re logster stuff [15:08:21] we shouldn't call that new metric ' requests' [15:08:32] as LineCountLogster could be used with any type of log file, not just access logs [15:08:40] how about [15:08:41] line_count [15:08:42] ? [15:12:41] ottomata: i see [15:12:58] ottomata: the problem is that line count when you see it in graphite is very non -descript [15:13:15] log_line_count [15:14:37] nuruia, you can use the prefix or suffix flags when you run the job, no? [15:14:56] https://github.com/wikimedia/operations-debs-logster/blob/master/logster/logster.py#L72 [15:15:02] nuria: ^ [15:15:11] ya ya, [15:15:13] so [15:15:16] ottomata: i am doing that [15:15:44] --metric-prefix='wikimetrics.request' would get you wikimetrics.request.line_count and .line_rate [15:16:10] maybe 'line_' is not good there, since you likely usually don't care about the number of lines, ja? [15:16:28] i'd even be fine if you wanted to change both of them to just 'count' and 'rate', and rely on folks using --metric-prefix [15:16:42] ottomata: ok, let's do that [15:16:45] ok [15:25:57] ottomata: isn't this going to mess up your icinga alarms andrew? cause it will change teh metric name [15:26:46] afaik, i don't use linecount logerster, do I? [15:27:10] AH CirrusSearch uses it [15:27:11] hm [15:27:32] right, looks like it is used somewhere in puppet [15:27:38] CirrusSearch-slow.log_line_rate [15:27:41] log_line_rate??? [15:27:45] i thoguht it was line_rate right now [15:27:46] h [15:27:47] hm [15:27:49] ahem ... [15:27:51] it is [15:27:59] ah [15:27:59] --metric-prefix CirrusSearch-slow.log [15:28:01] so [15:28:05] ahhhh [15:28:05] CirrusSearch-slow.log_line_rate [15:28:06] got it [15:28:07] i see [15:28:27] ok, cool, let's just change the puppet one to check for the new metric name then? [15:28:40] or, i mean, change the prefix to match the old one [15:28:42] so [15:28:49] --metric-prefix CirrusSearch-slow.log_line [15:28:50] haha [15:28:56] which will get us [15:29:01] ottomata: ya ya [15:29:02] CirrusSearch-slow.log_line_rate [15:29:04] :p [15:29:18] ottomata: k [15:35:59] ottomata: actually that would get us: --metric-prefix CirrusSearch-slow.log_line.rate [15:36:36] ottomata: right? as concatenation of the metric in graphite adds '.' [15:36:44] ottomata: let me see prefix choices [15:37:55] nuria: for ganglia [15:37:56] https://github.com/wikimedia/operations-debs-logster/blob/master/logster/logster.py#L165 [15:38:03] _ [15:38:11] graphite uses . [15:38:11] https://github.com/wikimedia/operations-debs-logster/blob/master/logster/logster.py#L202 [15:38:19] and this icinga chekc uses ganglia [15:38:24] k [15:42:39] ottomata: all changed now [15:49:41] cool looks good [15:49:51] will build deb [15:53:24] ottomata: ok, once build we can merge the rest of changes, since the line count is pretty handy i think i am going to add to wikimetrics a daily report count [15:53:34] aye [15:53:35] k [16:01:18] ottomata, nuria : deploying changes on refined webrequest table [16:01:22] nuria: where is the vanadium replacement ticket [16:01:25] joal: COOOOL [16:01:34] lemme know if you need any help [16:01:45] ottomata: https://phabricator.wikimedia.org/T90363 [16:01:52] Makes a few weeks I have not done, so I'll shout in any case ;) [16:02:02] ottomata: but that has no info as to the new boxes provisioned [16:02:37] ottomata: ori would know that [16:03:19] hm, ja, not sure why id din't see that in my board [16:03:20] danke [16:03:39] ottomata: just saw something -> when x_analytics is empty (contains '-'), then the map only contains {"-":null} [16:03:46] Not that god, eh ? [16:05:16] Analytics-Cluster, operations, Interdatacenter-IPsec: Secure inter-datacenter web request log (Kafka) traffic - https://phabricator.wikimedia.org/T92602#1162921 (Gage) p:Triage>Normal [16:07:11] HM [16:07:18] not really, true. [16:07:27] not that bad, but not that good [16:07:31] would be better if the map was null [16:07:34] is that possible to do? [16:07:48] probalby shoudl do that for the others too [16:07:57] It is (CASE WHEN) [16:08:04] k [16:08:15] user_agent case is managed by UDF, so we are covered [16:08:41] pushing the exit [16:08:46] oops :) [16:09:21] ok [16:09:48] nuria: http://apt.wikimedia.org/wikimedia/pool/main/l/logster/ [16:10:36] you can apt-get install this on wikimetrics host now [16:10:41] sudo apt-get install logster [16:10:42] ottomata: I see, this will be deployed by regular puppet run correct? [16:10:44] will upgrade it [16:10:45] no. [16:10:53] ottomata: ok, will do [16:10:59] ottomata: let me try now [16:12:57] hey nuria, [16:12:59] analytics.wikimetrics.ui.request_ [16:13:01] for prefix [16:13:03] question: [16:13:21] ottomata: i have changed it right? [16:13:40] ? [16:13:46] two questsions: [16:13:56] are you sure we want to prefix that with analytics/ [16:13:57] and [16:14:08] do you plan on having other metircs under ui. [16:14:09] ? [16:14:15] just wondering if that is necessary [16:14:16] that should be: analytics.wikimetrics.ui.request [16:14:30] Analytics-Kanban, Analytics-Visualization: Improve UX for VE/Wikitext comparison dashboard {lion} - https://phabricator.wikimedia.org/T94424#1162963 (kevinator) NEW [16:14:36] ottomata: Prefix with analytics in labs: yes, yuvi suggested that [16:14:40] ok [16:14:42] ah labs, right, ok. [16:15:04] ottomata: More metrics under ui: yes, I think i am going to include number of logins too [16:15:11] Analytics-EventLogging, Analytics-Kanban, Analytics-Visualization: Fully instrument editing experiences {epic} {lion} - https://phabricator.wikimedia.org/T89924#1162971 (kevinator) [16:15:11] ottomata: right, this is all labs [16:15:12] Analytics-Kanban, Analytics-Visualization: Improve UX for VE/Wikitext comparison dashboard {lion} - https://phabricator.wikimedia.org/T94424#1162972 (kevinator) [16:15:43] ottomata: lemme correct metric [16:15:57] ottomata: should be: analytics.wikimetrics.ui.request [16:16:07] ok, but why ui? [16:16:08] just curious [16:18:26] ottomata: because wikimetrics is two things: a web ui and a queue [16:19:03] ottomata: and "# of reports" would be analytics.wikimetrics.queue.reports.count [16:19:14] ottomata: but i can change it, np REALLY [16:19:22] ottomata: what would you think is best [16:20:20] hm [16:20:26] maybe put requests before ui [16:20:29] since they all fall under that [16:20:40] analytics.wikimetrics.requests.ui.count [16:20:57] analytics.wikimetrics.requests.reports.count [16:21:16] nuria: ^? [16:21:18] ottomata: but this code Only counts ui requests, not queue requests, as those do not go to apache [16:21:27] ah ok [16:21:31] that works too [16:21:49] that way you have a consitent tree in graphite, since all those count requsts [16:21:55] rather than forking above .request. [16:21:56] ottomata: sounds good [16:24:23] Analytics-Kanban, Analytics-Visualization: Improve UX for VE/Wikitext comparison dashboard {lion} - https://phabricator.wikimedia.org/T94424#1162995 (kevinator) [16:24:50] Analytics-EventLogging, Analytics-Kanban: Upgrade box for EventLogging (vanadium) - https://phabricator.wikimedia.org/T90363#1162997 (Ottomata) @ori, do you know the status of this? Nuria says that you know of some box that has been slated for this use? I'm happy to go ahead and do this upgrade, if you... [16:28:21] ottomata: corrected now [16:31:12] ah, nuria, your tab spacing is weird [16:31:17] jenkins is complaining [16:31:35] looks like a whole extra space column starting with the if? [16:34:01] ottomata: k , looking [16:42:30] ottomata: these other failures we do not care about, right? [16:42:45] https://integration.wikimedia.org/ci/job/operations-puppet-puppetlint-strict/17239/console [16:43:48] ja right [16:44:28] mreged nuria. i'll take care of the cirrus search one [16:44:28] danke. [16:44:37] ottomata: to you, mr [17:27:13] ottomata: yt? [17:30:49] yesm [17:31:55] nuria: ^ [17:32:41] ottomata: i swear this was working before ... but the line [17:32:42] if !$debug { [17:33:33] doesn't seem to work , the crontab doesn't update [17:33:47] when running puppet agent -tv [17:34:00] hm [17:34:51] hm, nuria, not sure. what I do when this happens is do some hacky output debugging [17:34:55] this is a sefl hosted puppet master, right? [17:34:59] do something like [17:34:59] yes [17:35:20] file { '/tmp/debugit': content => inline_template("wikimetrics_debug: ${debug}") } [17:35:21] somewhere [17:35:23] or [17:35:24] actually [17:35:28] you should be able to use notice [17:35:39] http://www.puppetcookbook.com/posts/simple-debug-messages.html [17:35:59] or notify [17:36:26] ok [17:36:31] will try this [17:43:09] ergh, need some node JS help [17:52:07] ottomata: double-checked on real data, looks good now [17:52:20] Pushing the correction [17:53:19] awesoome [17:53:19] k [17:58:51] (PS1) Joal: Correct refined webrequest to have null x_analytics_map instead of not meaningfull one. [analytics/refinery] - https://gerrit.wikimedia.org/r/200623 [18:17:57] joal, gotta a trailing space there, otherwise looks good. [18:18:09] joal: just in case, what happens if x_analytics itself is NULL [18:18:11] ? [18:18:26] ottomata: COALESCE ;) [18:19:02] ? [18:20:30] In SQL, coalesce is an easy way to convert NULL to other values [18:20:50] ottomata: --^ [18:20:59] (CR) Nuria: [C: 2] Correct refined webrequest to have null x_analytics_map instead of not meaningfull one. (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/200623 (owner: Joal) [18:21:28] ottomata: Here, COALESCE(x_analytics, '-') means: x_analytics, or '-' if it is null [18:21:35] yes [18:21:38] oh [18:22:06] oh! [18:22:15] so if null, use -, and that will case to NULL anyway [18:22:16] ok. [18:22:17] cool [18:22:22] :D [18:22:35] k fix trailing space and I merge :) [18:22:43] line 83 [18:23:21] (PS2) Joal: Correct refined webrequest to have null x_analytics_map instead of not meaningfull one. [analytics/refinery] - https://gerrit.wikimedia.org/r/200623 [18:46:34] ottomata: mergy mergy ? [18:51:55] (CR) Ottomata: [C: 2 V: 2] Correct refined webrequest to have null x_analytics_map instead of not meaningfull one. [analytics/refinery] - https://gerrit.wikimedia.org/r/200623 (owner: Joal) [18:51:55] cone [18:51:55] done [18:52:02] Thx [18:53:50] Analytics-Wikimetrics, Community-Wikimetrics: Plain language definitions of Wikimetrics metrics - https://phabricator.wikimedia.org/T93685#1163969 (Capt_Swing) [18:54:05] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Get a measure of daily usage of wikimetrics by userbase - https://phabricator.wikimedia.org/T94193#1163970 (Capt_Swing) [18:54:17] Analytics-Wikimetrics, Community-Wikimetrics: Description of metrics includes link to on-wiki metrics documentation - https://phabricator.wikimedia.org/T93659#1163971 (Capt_Swing) [18:54:28] Analytics-Wikimetrics, Community-Wikimetrics: some non-Latin characters do not show up in uploaded usernames and result in invalid usernames - https://phabricator.wikimedia.org/T93646#1163977 (Capt_Swing) [18:54:41] Analytics-Wikimetrics, Community-Wikimetrics: Story: WikimetricsUser searches for cohort (filters) using tag name - https://phabricator.wikimedia.org/T75071#1163978 (Capt_Swing) [18:54:53] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Story: WikimetricsUser reads user names in a JSON report [8 pts] - https://phabricator.wikimedia.org/T74747#1163981 (Capt_Swing) [18:55:03] Analytics-Wikimetrics, Community-Wikimetrics: Give the option of using the same time range for all reports for a given cohort - https://phabricator.wikimedia.org/T74117#1163989 (Capt_Swing) [18:55:14] Analytics-Wikimetrics, Community-Wikimetrics: Story: Wikimetrics user views full list of cohort members - https://phabricator.wikimedia.org/T86335#1163991 (Capt_Swing) [18:55:24] Analytics-Wikimetrics, Community-Wikimetrics: Excessive failure messages trying to create cohorts and run messages on Wikimetrics - https://phabricator.wikimedia.org/T87235#1163993 (Capt_Swing) [18:55:35] Analytics-Wikimetrics, Community-Wikimetrics: Uploading cohort or running a large report fails - https://phabricator.wikimedia.org/T87596#1163994 (Capt_Swing) [18:55:47] Analytics-Wikimetrics, Community-Wikimetrics: Story: Wikimetrics User uses tooltip to view all projects for an expanded cohort member - https://phabricator.wikimedia.org/T86332#1163996 (Capt_Swing) [18:55:56] Analytics-Wikimetrics, Community-Wikimetrics: Make timezone selector comprehensive and consistent - https://phabricator.wikimedia.org/T88604#1163998 (Capt_Swing) [18:56:06] Analytics-Engineering, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Story: WikimetricsUser reports pages edited by cohort - https://phabricator.wikimedia.org/T75072#1164000 (Capt_Swing) [18:57:18] Analytics-Wikimetrics, Community-Wikimetrics: Story: user wants to be able to re-run a failed report more easily - https://phabricator.wikimedia.org/T88610#1164015 (Capt_Swing) [19:02:27] ottomata: one double check before breaking everything ;) [19:02:33] yes? [19:02:53] (CR) Fhocutt: [C: 1] "It's strange you were getting an exception where I wasn't; I'm using vagrant, and `pip freeze` gives me `Flask==0.10.1`." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/199814 (https://phabricator.wikimedia.org/T93023) (owner: Fhocutt) [19:02:54] ottomata: I'll only have to stop/restart the ozzie job for webrequest refinement, right [19:03:33] Analytics-Kanban, Community-Wikimetrics, Patch-For-Review: Get a measure of daily usage of wikimetrics by userbase - https://phabricator.wikimedia.org/T94193#1164047 (Capt_Swing) [19:03:38] Th rest should be left untouched, even if we deploy [19:03:49] Analytics-Kanban, Analytics-Wikimetrics, Patch-For-Review: Get a measure of daily usage of wikimetrics by userbase - https://phabricator.wikimedia.org/T94193#1157164 (Capt_Swing) [19:05:41] joal: yes [19:05:45] that is correct [19:05:50] Analytics-Kanban, Analytics-Wikimetrics, Community-Wikimetrics, Patch-For-Review: Utf-8 names on json reports appear as unicode code points: "\u0623\u0645\u064a\u0646" - https://phabricator.wikimedia.org/T93023#1164053 (Capt_Swing) [19:06:01] ottomata: Thx, I just wanted reassurance here :) [19:06:12] Let's go then :) [19:07:41] ok! [19:13:27] Analytics-Engineering, Analytics-Wikimetrics, Community-Wikimetrics: User reads result of validation after creating a cohort - https://phabricator.wikimedia.org/T76914#1164085 (Capt_Swing) [19:16:35] Analytics, Analytics-Kanban, Patch-For-Review: Turn off WP Zero's Limn-Dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1164098 (kevinator) \o/ [19:19:10] Analytics-Wikimetrics, Community-Wikimetrics: some non-ascii characters do not show up in uploaded usernames and result in invalid usernames - https://phabricator.wikimedia.org/T93646#1164113 (Fhocutt) [19:26:37] this is EEVS, not Wikimetrics, correct? [19:26:38] https://phabricator.wikimedia.org/T70477 [19:28:25] Analytics-Wikimetrics, Community-Wikimetrics: Give the option of using the same parameters for all reports for a given cohort - https://phabricator.wikimedia.org/T74117#1164167 (Capt_Swing) [19:28:39] nuria, fhocutt: I had a look at https://gerrit.wikimedia.org/r/#/c/199814/2 and it looks good. Passed the tests and checked it worked on vagrant. May I merge it? [19:29:03] Analytics-Wikimetrics, Community-Wikimetrics: Give the option of using the same parameters for all reports for a given cohort - https://phabricator.wikimedia.org/T74117#772339 (Capt_Swing) I updated this task with more detail, and generalized it to describe all settings where there is currently redundancy. [19:31:04] Hi, I've been looking around at wikimedia analytics tools. How do I get the stats on VisualEditor usage? I know I can get the recent changes unto last 500 edits. [19:31:15] Analytics-Wikimetrics, Community-Wikimetrics: Plain language definitions of Wikimetrics metrics - https://phabricator.wikimedia.org/T93685#1164178 (Fhocutt) a:Fhocutt [19:31:19] What if I more data? [19:31:53] (CR) Mforns: [C: 2 V: 2] "LGTM" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/199814 (https://phabricator.wikimedia.org/T93023) (owner: Fhocutt) [19:31:58] Or say answer questions like, how many people switched from native editor to VisualEditor ? [19:32:46] Analytics-Wikimetrics: Story: WikimetricsUser runs report against all wikis - https://phabricator.wikimedia.org/T70477#1164182 (Fhocutt) Is this Wikimetrics or EEVS? If not Wikimetrics, it shouldn't be on this board. [19:33:11] *upto [19:33:32] *what if I want more data [19:33:33] Analytics-Wikimetrics, Community-Wikimetrics: Description of metrics includes link to on-wiki metrics documentation - https://phabricator.wikimedia.org/T93659#1164186 (Fhocutt) a:Fhocutt [19:34:18] Analytics-Wikimetrics, Community-Wikimetrics: Plain language definitions of Wikimetrics metrics - https://phabricator.wikimedia.org/T93685#1164192 (Fhocutt) [19:34:19] Analytics-Wikimetrics, Community-Wikimetrics: Description of metrics includes link to on-wiki metrics documentation - https://phabricator.wikimedia.org/T93659#1142792 (Fhocutt) [19:34:48] Analytics-Engineering, Analytics-Wikimetrics, Community-Wikimetrics: User reads result of validation after creating a cohort - https://phabricator.wikimedia.org/T76914#1164197 (Fhocutt) a:Fhocutt [20:07:58] nuria: yt ? [20:14:02] nuria: I’ve finally moved to SF btw [20:19:45] YuviPanda, welcome! :D [20:19:51] fhocutt: \o/ [20:24:45] heya, can anyone help me with a nodejs thang? [20:25:12] I am not an expert, but I can try :) [20:26:33] ok so [20:26:40] i am trying to read from rcstream and produce to kafka [20:26:44] here is the example: [20:27:03] https://wikitech.wikimedia.org/wiki/RCStream#JavaScript [20:27:07] so, that works. [20:27:13] but, now I am trying to produce to kafka [20:27:39] http://www.codeshare.io/aITwc [20:27:56] and, nothing happens [20:28:03] in fact, i believe (double checking now) [20:28:06] that, if I do [20:28:09] var kafka = require('kafka-node'), [20:28:23] and then do the original example, where I just do console.log(data) [20:28:26] it also doesn't work [20:29:41] So just adding the kafka var would break the rcstream thing ? [20:30:02] yes [20:30:11] weeeeeeeeirdo ! [20:30:22] yeah, just double checked [20:30:25] pasted in what I am trying now [20:31:27] i see milimetric online, is he really there? [20:31:38] yeah, just nothing happens [20:31:44] as if no on change event fires [20:32:24] nuria: ? [20:32:41] hmm [20:33:02] The connection happens though ? [20:34:05] connection with rcstream? [20:34:09] i dunno, i mean, i get no errors [20:34:14] also, i can produce to kafka manually [20:36:17] the return from both socket.on calls in each case is the same [20:41:36] otto Man ... [20:41:57] joal: if I do the kafka require first. [20:41:58] it works! [20:42:15] Mouarf [20:42:28] I would have tried as well, but it's bizarre [20:43:08] ottomata: deployed finished and checked [20:43:15] aweSOOOME [20:43:24] Doc updated [20:43:25] we got those fields rolling in during refinement? [20:43:27] so cool [20:44:02] Analytics, Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1164596 (JAllemandou) Open>Resolved [20:44:29] Analytics-Cluster, Analytics-Kanban: Add x_analytics map to refined tables - https://phabricator.wikimedia.org/T89396#1164607 (JAllemandou) Open>Resolved [20:44:30] Analytics-Cluster, Analytics-Kanban, Performance: Implement Unique Clients report on cluster using x-analytics header & last access date {bear} - https://phabricator.wikimedia.org/T92977#1164609 (JAllemandou) [20:44:31] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Mobile PMs has reports on session-related metrics from Wikipedia Apps - https://phabricator.wikimedia.org/T86535#1164608 (JAllemandou) [20:44:50] Analytics-Kanban, Analytics-Visualization: Mobile PMs has visualization on session-related metrics from Wikipedia Apps - https://phabricator.wikimedia.org/T94481#1164611 (kevinator) NEW [20:44:54] Analytics-Cluster, Analytics-Kanban: Add user_agent map to refined tables - https://phabricator.wikimedia.org/T91793#1164620 (JAllemandou) Open>Resolved [20:44:55] Analytics-Cluster, Analytics-Kanban, Performance: Implement Unique Clients report on cluster using x-analytics header & last access date {bear} - https://phabricator.wikimedia.org/T92977#1125610 (JAllemandou) [20:44:56] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Mobile PMs has reports on session-related metrics from Wikipedia Apps - https://phabricator.wikimedia.org/T86535#1164621 (JAllemandou) [20:45:01] Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Mobile PMs has reports on session-related metrics from Wikipedia Apps - https://phabricator.wikimedia.org/T86535#1164625 (kevinator) [20:45:30] Now the thing will be to define what we mean by "sanitized" [20:45:42] And currate the format :) [20:46:23] ottomata: Sorry for my no-help on js ... [20:47:23] joal: which field needs to be "sanitized"? [20:47:44] Well, not the fields per se, but the table [20:48:04] ip address, user agent, mainly [20:48:05] Analytics-Kanban, Analytics-Visualization: Mobile PMs has visualization on session-related metrics from Wikipedia Apps - https://phabricator.wikimedia.org/T94481#1164655 (kevinator) [20:48:08] For instance: are we hapy to give to the community the table as-is, or do we want to remove specific daat [20:48:11] and geocoded_data i guess [20:48:11] ... [20:48:14] * Ironholds blinks [20:48:19] no, we are not happy to do that. [20:48:21] haha, Ironholds [20:48:30] this table is not slated for community as is :p [20:48:37] oh good [20:48:41] because there is SO MUCH stuff there. [20:48:55] That's why I said sanitized :) [20:49:25] And maybe there will never be enough sanitixation for us to release to the community ... [20:49:33] if you were going row by row [20:49:41] you'd need to kill all geodata, ip, x_forwarded, user agent, x_analytics, uri_query [20:49:45] i think the table is too big to release to community anyway, joal. we need aggregates [20:49:55] yup [20:50:04] ok, sounds good to me [20:50:28] I had misunderstood something somewhere :) [20:50:45] right now, a higher priority would be to release PV data with the new definition [20:51:16] aye [20:51:30] that is pretty trivial to do. I could put it together with otto in a coupla hours. [20:51:57] we just need an oozie job I've already scoped in my brain and then to stealie the rsync code from the existing hive-to-dumps.wikimedia script. [20:52:02] * Ironholds shrugs [20:52:46] yeah, let's scope out a project out first with the deliverables documented. [20:52:56] I also want this to go into vital signs [20:53:13] well, a better first step would be "let's fix the known bugs in the pageview definition" [20:53:13] visualization is part of the deliverable [20:53:20] and then talk about getting that data [20:53:32] I'll let you drive kevinator , and try to execute ;) [20:54:00] Let me start with a new task :-) [20:54:06] loggin one now [20:54:51] Ironholds: are the known issues with the current PV definition documented? [20:54:59] i.e. in phab? [20:55:06] yes! [20:55:39] and have been documented, explicitly called out via email, and neither prioritised nor scheduled, since they were reported two and a half weeks ago [20:56:20]