[00:00:37] kaldari: we have 14K responses for version (a) but 13K taskTokens. [00:02:20] leila: Do you mean 13K unique taskTokens or 13K taskTokens total? [00:03:19] yeah, that's one problem, kaldari. there are duplicate rows. Ellery and I will look more. will ping you if something interesting comes out of oit [00:03:25] *it [00:03:35] leila: I’m not sure how much the randomness bug affected the uniqueness, but nuria probably knows more. [00:04:27] Analytics, Analytics-Kanban, Patch-For-Review: Turn off WP Zero's Limn-Dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1134133 (QChris) [00:04:39] Analytics, Analytics-Kanban, Patch-For-Review: Turn off WP Zero's Limn-Dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1123876 (QChris) [00:05:47] kevinator: Can I leave the communication with df-oy and the asking whether the pages look like they intended to you? [00:06:19] qchris: sure, I'll ask him now. [00:06:22] Cool. [00:07:32] Analytics, Analytics-Kanban, Patch-For-Review: Turn off WP Zero's Limn-Dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1134144 (QChris) [00:12:38] qchris: dfoy already left for the day. I sent him an email and should hear from him tomorrow [00:12:52] I'm heading home too... [00:12:56] Thanks again! [00:13:08] kevinator: Sure. Whenever he signed off. Please just check the corresponding box on the task. [00:13:14] Thanks. [00:13:17] Have a nice evening. [00:14:24] Analytics, Analytics-Kanban, Patch-For-Review: Turn off WP Zero's Limn-Dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1134168 (QChris) [00:44:14] Analytics-Cluster, Analytics-Kanban, Performance: Implement Unique Clients report on cluster using x-analytics header & last access date {bear} - https://phabricator.wikimedia.org/T92977#1134248 (csteipp) >>! In T92977#1134063, @kevinator wrote: > I merged in T88814 because it's the same work, but that... [00:49:16] (PS1) Milimetric: [WIP] Create A/B comparison modules [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [01:16:31] Analytics-Cluster, FINCH, Wikimedia-General-or-Unknown: Browser and platform stats for logged-in vs. anon users for security and product support decisions - https://phabricator.wikimedia.org/T58575#1134289 (Nuria) [03:12:57] (PS5) OliverKeyes: De-static-everything [analytics/refinery/source] - https://gerrit.wikimedia.org/r/197296 [03:13:12] (CR) OliverKeyes: "That should address the naming issues :)" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/197296 (owner: OliverKeyes) [12:01:24] Analytics-Cluster, Analytics-Kanban, Easy: Mobile Apps PM has monthly report from oozie about apps uniques [8 pts] - https://phabricator.wikimedia.org/T88308#1135077 (JAllemandou) This report is run monthly, the first of each month, on previous month data. It updates the data file adding newly computed... [14:57:05] Analytics, Analytics-Cluster: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1135417 (Ottomata) p:Triage>Normal a:Ottomata [14:57:21] Analytics, Analytics-Cluster: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1095420 (Ottomata) Sounds fun! I wonder if there are any other fields we want to add. If I do this, it would be easier to do them all at the same time. [15:01:21] (CR) Ottomata: Move UAParser wrapper to refinery-core and update refinery-hive accordingly. (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/195952 (owner: Joal) [15:06:14] Analytics-EventLogging: Adapt eventlogging intake to use Kafka - https://phabricator.wikimedia.org/T93096#1135469 (Ottomata) Open>Resolved /beacon/event.gif works! :) [15:06:41] (PS3) Joal: Move UAParser wrapper to refinery-core and update refinery-hive accordingly. [analytics/refinery/source] - https://gerrit.wikimedia.org/r/195952 [15:07:32] Analytics-Engineering, Analytics-EventLogging: Researchers access EventLogging logs to troubleshoot new experiments - https://phabricator.wikimedia.org/T85027#1135473 (Ottomata) Can I decline this ticket? Orrrrrrrrr what should I do with it? It is one of very few high priorities assigned to me, and it do... [15:22:20] Analytics-Cluster: Add done-flag for webrequest partitions indicating that the partition is available (but not yet checked) - https://phabricator.wikimedia.org/T85811#1135508 (Ottomata) Open>Resolved Done a long time ago! [15:52:50] Analytics, Analytics-Cluster: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1135565 (Ironholds) What is available? ;) [16:19:16] Analytics, Analytics-Cluster: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1135635 (Ottomata) Basically this: https://www.varnish-cache.org/docs/3.0/reference/varnishncsa.html [16:24:59] Analytics, Analytics-Cluster: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1135653 (Ironholds) Oh god we're using the CLF? That thing haunts my nightmares. More seriously: I think I'm fine with what we've got, absent X-Cache :) [16:31:07] (CR) Nuria: [C: 2] "I think we should decide on whether we are using "_" for variable naming, so webrequestInstance? (camelCase) or webrequest_instance (I pre" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/197296 (owner: OliverKeyes) [16:31:17] (CR) Nuria: [V: 2] "I think we should decide on whether we are using "_" for variable naming, so webrequestInstance? (camelCase) or webrequest_instance (I pre" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/197296 (owner: OliverKeyes) [16:32:04] nuria, thanks! I will now make a second patch for mforns_brb 's use case :) [16:32:37] Ironholds: ok, for sessions? [16:32:41] (PS2) Milimetric: [WIP] Create A/B comparison modules [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [16:33:04] Ironholds: what does he need from these UDFS again? (sorry) [17:05:00] nuria: you are trying to fix the reporter thing? [17:05:05] i can see why it is broken [17:05:19] ottomata: just did, was about to send you a CR [17:05:23] ah cool [17:05:23] ok [17:05:24] ottomata: no, no worries [17:05:37] ottomata: CR on the way [17:05:38] you using get_reader instead of the manual zmq socket creation somehow then? [17:05:39] k [17:05:40] cool [17:06:07] ottomata: no, is even easier than that , it just is not 'generic' in any way [17:06:12] k [17:33:20] milimetric, hey [17:53:39] (CR) Nuria: [C: 1] "I think this is ready to merge, let me know otherwise." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/195952 (owner: Joal) [17:58:40] hey mforns, just back from lunch [17:58:46] hi! [17:58:58] how can I help [17:59:19] just wondering.. I found some old reports that have 'PENDING' status [17:59:25] for a long time now, days.. [17:59:36] I suppose those are not running any more, right? [17:59:52] yea [18:00:05] ok [18:00:12] the pending status is a sign of a dead report [18:00:21] because there's a hard time limit on the celery task [18:00:33] so any more than I think a few hours and it'll get killed [18:00:43] Is there another way to see the reports that are being processed without looking at the db? [18:00:44] ok [18:03:01] mforns: not really... you could look through the celery queue logs I guess [18:03:17] there are different messages for reports starting, succeeding, failing, etc. [18:04:15] milimetric: back to limn1 after fixing el [18:04:17] aha, and maybe the celery cli? [18:04:23] milimetric: there are local chnages on box: [18:04:27] https://www.irccloud.com/pastebin/63VzBj6U [18:05:24] mforns: the celery cli will give you task ids, yes, which are not reports, 1report is normally many task ids [18:05:39] nuria, oh, ok [18:06:26] nuria / mforns: not really actually, because RunReport is the only thing that spawns a celery task now [18:07:10] so yeah, celery cli would help too, but i've never used it for more than basic debugging [18:07:30] nuria: yeah, that was my old change that is on puppet but somehow if you run puppet it doesn't get applied [18:07:45] so when I did it, I manually applied it, and left the commit there so you could take a look when you started [18:07:46] nuria: +2 except flake8? [18:07:48] milimetric: wait ...more things that runreport spawn tasks, like cohort validation , right? [18:08:12] ottomata: note that flake8-bin doesn't report errors is just plain not running, right? [18:09:12] you're right nuria, true, there are other kinds of tasks. just no other reports (cc mforns) [18:09:39] nuria, it does [18:09:39] 17:33:37 bin/eventlogging-reporter:89:5: E265 block comment should start with '# ' [18:09:41] right? [18:09:46] milimetric, ok [18:10:03] i think it wants a space after the #? [18:10:26] ottomata: argh wait , i did not checked after i added the TODO, you are right [18:10:59] ottomata: my local tox -e flake8 doesn't complain though, correcting now [18:12:57] ottomata: Sorry gain, corrected now [18:13:00] *again [18:24:50] nuria, merged. [18:25:12] ottomata: k, i fixed stuff on the box, so i will to deploy that [18:26:02] k [18:29:43] Analytics-EventLogging, Analytics-Kanban, operations: EventLogging query strings are truncated to 1014 bytes by ?(varnishncsa? or udp packet size?) - https://phabricator.wikimedia.org/T91347#1136154 (Ottomata) So, we have implemented a varnishkafka eventlogging endpoint that works well and dandy. But!... [18:34:37] Analytics, Analytics-Cluster: Log the X-Cache header in the webrequest logs - https://phabricator.wikimedia.org/T91749#1136184 (Ottomata) And these: HTTP/1.1 200 OK Server: Apache X-Content-Type-Options: nosniff X-Analytics: page_id=1093068;ns=0 Content-language: en X-UA-Compatible: IE=Edge Vary: Accept... [18:35:35] mforns, working on your patch now [18:35:51] hye Ironholds, my patch? :] ok! [18:36:14] the "we need isAppPageview as a public method" problem [18:36:17] thanks! I'm troublehooting wikimetrics now, so no pressure [18:36:31] of course, it's now complicated by the fact that we need the apps team to make app views actually a thing, but... :/ [18:37:11] yes.. [18:37:49] so, Ironholds I can still start working on the task without you having finished this, so yea [18:37:53] thanks! [18:38:07] mforns, cool! [18:38:17] the definition will be somewhat broken until apps gets their stuff together, note [18:53:42] Analytics, Analytics-Kanban, Patch-For-Review: Turn off WP Zero's Limn-Dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1136225 (kevinator) [18:55:57] Analytics, Analytics-Kanban, Patch-For-Review: Turn off WP Zero's Limn-Dashboards & put up a "moved sign" - https://phabricator.wikimedia.org/T92920#1136240 (kevinator) dfoy had a look at the pages where the dashboards used to show up up and said it looked good. [18:57:23] (PS3) Milimetric: [WIP] Create A/B comparison modules [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [18:57:40] funnel part done, with all filters, \o/ [19:09:10] Analytics-Kanban, Analytics-Wikimetrics: Troubleshoot Wikimetrics RAE reports - https://phabricator.wikimedia.org/T93217#1136309 (mforns) Two things: 1) Regarding previous comment, I've executed RAE report with the cohort in question, and it worked fine. So the problem seems to exist only for RNAE. 2) A... [19:11:44] milimetric: then on lim1 instance there are two local commits: [19:12:06] milimetric: [19:12:12] https://www.irccloud.com/pastebin/sAaucFNn [19:12:14] nuria: yeah, one is the "official" one that sets up the other dashboards [19:12:27] and another one is mine, which should be fixed up and merged into the "official" one probably [19:12:29] milimetric: why does that have to be local to the box? [19:12:50] milimetric: is there anything private about it [19:12:51] how would you configure it otherwise? [19:13:12] the problem is that the configuration is for N dashboards [19:13:14] can't it be on puppet depot linked to the particular labs instance? [19:13:45] and I think last time ottomata tried, he couldn't figure out a way to pass an "array" or "set" of things to puppet from a config variable set in the instance's config [19:14:13] nuria: right, but it's not a static setup [19:15:29] yes! [19:15:30] you an now [19:15:31] can [19:15:34] hiera [19:16:11] milimetric: https://wikitech.wikimedia.org/wiki/Hiera:Analytics [19:16:18] milimetric, ottomata : ok, so what is what we are trying to do here? setup an apache instance to server dashiki/some static files? [19:16:23] nuria / ottomata: just remember, this is due next week, with logster and everything working [19:16:31] so try not to refactor too much if you don't have to [19:16:33] i dunno what yall are even talking about :) [19:16:43] hangout? [19:16:44] milimetric: it's not happening next week i can tell you that [19:16:55] nuria: ok, then I'll just do it manually, you don't have to do it [19:18:55] milimetric: the manual approach doesn't seem teh best, as all logster config will also need to be manual and just deployed to the box ony [19:19:35] I'm not sure I want to get into a holy war about puppet or anything. But this is why I find the whole idea so frustrating [19:19:43] because this task is something that takes minutes to do manually [19:19:51] and yet we've been staring at it on our board for three months [19:20:24] anyway: really, I've got to focus on the dashboard, I'm really really close to finishing and I really want to finish in the next hour. [19:20:47] so you're free to drop the task, I don't mind, I'll just do it Monday, I'm sure there's a way I can do it and it'll all be fine [19:21:22] milimetric: well, if it creates more technical debt is really not fine [19:21:51] milimetric: but we can talk about it at a later time, doesn't seem that this belongs on lim1 at all [19:22:09] ok, I'll make my own instance and set it up there [19:31:15] milimetric: can't this be like dashiki and be served by wikimetrics apache? [19:31:46] milimetric: seems a lot simpler as it doesn't require anything from limn [19:31:54] there are no restrictions on where / how it can be served except it would be nice if http://edit-analysis.wmflabs/ would point to it [19:32:03] nuria: why do we require anything from limn?! [19:32:18] limn has nothing to do with this, it's just the name of a box [19:32:21] milimetric: cause all stuff on limn1 sets a fallback from apache to node [19:32:43] sure, wikimetrics sets up its own stuff on the instances it runs on as well [19:32:50] milimetric: right, not lim, node [19:33:08] it's up to you, really, do anything you see fit [19:33:41] the deadline is next week, as I said - I can just restore my manual setup on limn1 if you aren't done by then [19:41:58] milimetric, mforns, nuria: I have updated my slide deck. feel free to comment on it. I think we should leave our goal to same as this qtr. I added aspirational activities... but there's a lot of concurrent work we'll be taking on. That part is definitely up for discussion & maybe even some high level release planning [19:42:06] milimetric, mforns, nuria: https://docs.google.com/a/wikimedia.org/presentation/d/1YvswGSk7JWPbfshf3YVTLnAbG1_RQikHLZZ8sxmlX1Q/edit?usp=sharing [19:42:32] I need to get lunch and will be back in an hour if you want to chat in the batcave [19:43:32] kevinator, ok! [19:46:37] milimetric: if you prefer limn1 and you want the http://edit-analysis.wmflabs/ I will restore that, I can do that too, i just think for stuff like this going forward we should not create more technical debt. [19:48:56] kevinator: looks good, I think we need to distill one goal from that if that is what we are asked to do [19:50:00] nuria: I also don't like technical debt, but priority #1 is that this gets done next week. I'm not saying create technical debt if we can avoid it. [19:55:23] milimetric: well, priority #1 is keeping lights up, other things come after. [19:57:50] Analytics-Kanban, Analytics-Wikimetrics: Troubleshoot Wikimetrics RAE reports - https://phabricator.wikimedia.org/T93217#1136531 (mforns) Well I executed the critical projects (with the big queries) separately and they went fine, so yea, forget it. [20:01:50] hi kaldari. are we meeting now? [20:02:02] I'm in the Hangout, but no one is there kaldari. [20:02:32] Analytics-EventLogging: Make eventlogging-reporter work with generic processor uris, not just 0mq sockets. - https://phabricator.wikimedia.org/T93415#1136542 (Ottomata) NEW a:Ottomata [20:03:16] Eloquence: are you in the meeting? [20:03:27] no one is in the Hangout Eloquence. [20:03:46] Analytics-Cluster, Analytics-Kanban, Performance: Implement Unique Clients report on cluster using x-analytics header & last access date {bear} - https://phabricator.wikimedia.org/T92977#1136554 (Anomie) >>! In T92977#1134248, @csteipp wrote: > @Nuria, cookies live for as long as we tell them to. What'... [20:05:28] leila, they're setting it up right now [20:05:45] thanks much Eloquence. [20:07:42] Analytics-Cluster, Analytics-Kanban, Performance: Implement Unique Clients report on cluster using x-analytics header & last access date {bear} - https://phabricator.wikimedia.org/T92977#1136572 (Nuria) @csteipp: I was planning to let cookie live forever, its value is updated upon access once a day. [21:15:11] Analytics-Cluster, Analytics-Kanban, Performance: Implement Unique Clients report on cluster using x-analytics header & last access date {bear} - https://phabricator.wikimedia.org/T92977#1136804 (Nuria) @csteipp: Having the cookie live forever means that no date calculations are needed on VCL for an ex... [21:32:43] (PS4) Milimetric: Implement A/B comparison [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [21:56:59] Analytics-Cluster, Analytics-Kanban, Performance: Implement Unique Clients report on cluster using x-analytics header & last access date {bear} - https://phabricator.wikimedia.org/T92977#1137020 (kevinator) @csteipp, as the Product Manager for analytics, I am driving this requirement. Monthly and Dail... [22:13:15] (PS5) Milimetric: Implement A/B comparison [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [22:18:53] (PS6) Milimetric: Implement A/B comparison [analytics/dashiki] - https://gerrit.wikimedia.org/r/198169 [22:28:29] Analytics-Cluster, Analytics-Kanban: Add x_analytics map to refined tables - https://phabricator.wikimedia.org/T89396#1137163 (kevinator) [22:28:30] Analytics-Cluster, Analytics-Kanban: Add user_agent map to refined tables - https://phabricator.wikimedia.org/T91793#1137164 (kevinator) [22:28:31] Analytics-Cluster, Analytics-Kanban: Mobile PMs has reports on session-related metrics from Wikipedia Apps - https://phabricator.wikimedia.org/T86535#1137162 (kevinator) [22:51:16] milimetric: the edit site is restored on limn1: https://edit-analysis.wmflabs.org/ [22:51:23] cool, thanks :) [22:51:38] dashboard's almost done, but I don't have real data yet, so I won't publish anything [22:51:41] milimetric: will do a commit of the stuff that doesn't to be on the box [22:51:59] k, that works [22:52:56] milimetric: but i am not going to add logster by hand to the limn1.pp local file, more so when there is no pageviews to count as there is no data to see. [22:53:14] milimetric: will keep working on logster changes on wikimetrics [22:53:50] nuria: ok [22:54:00] (personally I never really thought the whole page counting thing was all that important) [22:54:30] milimetric: i just would like to get away from changes deployed locally to one box as much as possible [22:54:54] milimetric: will do puppet commits later and send CR to otto [23:32:15] Analytics, MediaWiki-Core-Team, VisualEditor, VisualEditor-Performance, and 3 others: Apply Schema:Edit instrumentation to WikiEditor - https://phabricator.wikimedia.org/T88027#1137389 (Jdforrester-WMF) [23:35:36] Analytics-Visualization, Patch-For-Review: Limn does not show metrics for WikiData - https://phabricator.wikimedia.org/T60450#1137431 (Aklapper) Open>Resolved a:Aklapper Patch was merged long ago; closing as resolved. Please reopen if there's more to fix here.