[00:12:22] (CR) OliverKeyes: "It would be nice, but I'm not entirely sure how we'd do it. We could push some of the logic upstream and have it spit out the result in a " [analytics/refinery/source] - https://gerrit.wikimedia.org/r/193985 (owner: OliverKeyes) [00:16:01] (CR) OliverKeyes: "Alternately: if we get the host extractor included in the ETL stage and modify it to spit out ['project_variant'=String,'project_class'=St" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/193985 (owner: OliverKeyes) [00:17:44] (CR) OliverKeyes: "Oh! And getting the apps team to spit out pageid/ns in the same way MediaWiki does." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/193985 (owner: OliverKeyes) [00:33:47] Analytics, Wikimedia-Fundraising: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1080580 (atgo) p:Triage>Normal [00:41:52] Analytics-EventLogging: Some events not validating for MultimediaViewerNetworkPerformance - https://phabricator.wikimedia.org/T91347#1080600 (mforns) NEW [00:49:24] nuria: how long does it take for EL events to arrive to analytics-store? [00:57:42] tgr: to the logs? [00:57:53] tgr: the logs a rsyn-ed once a day [00:58:07] tgr: The db master receives events right away [00:58:11] tgr: makes sense? [00:59:04] I meant the DB [01:00:09] it's an unrelated question, I just deployed some logging code and the corresponding table does not exist and I want to know whether I should be worried [01:00:28] which DB is the master? [01:01:30] tgr: if you just deployed the code 1st thing to look at would be whether events are coming in [01:01:38] tgr: what event is it? [01:02:04] nuria, I'm going to review the patch now, sorry for the delay [01:02:16] eh, nevermnind, I forgot beta sends events to a different place [01:02:23] it's not on production yet [01:02:32] ori: np, i am planning on testing it on vanadium with some backfilling today [01:03:40] tgr: but we can look in beta labs, wnat to? [01:03:55] tgr: https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs [01:04:06] nuria: I'll do that, thanks [01:04:31] tgr: you are going to need sudo in the machine, if you do not have it just request it and i will look for ya [01:05:35] tgr: let me know, i will be here for a while [01:06:38] Analytics-EventLogging: Some events not validating for MultimediaViewerNetworkPerformance - https://phabricator.wikimedia.org/T91347#1080671 (Tgr) Image is a CORS request, all others (with the possible exception of userinfo) are local AJAX requests, so the headers are different due to CORS restrictions (and... [01:07:28] Analytics-EventLogging, Analytics-Kanban: Some events not validating for MultimediaViewerNetworkPerformance - https://phabricator.wikimedia.org/T91347#1080672 (Nuria) [01:08:31] re the schema validation error, I imagine there is a server-side validation as well [01:08:49] is it possible that it's not completely in sync with the client-side? [01:09:14] tgr: are we talking about your new events? [01:09:39] no, sorry, topic-jumping [01:09:42] about T91347 [01:10:20] tgr:you know... i suspect nothing to do with EL or js, more like encoding [01:10:51] oh, yeah, there is an "Unable to decode" error, I see now [01:11:16] tgr: that is url decoding but you know... url decoding has to happen before json validation [01:11:44] still weird that such a low-level error would only affect certain schemas [01:12:08] tgr: That one - guessin here- is a problem decoding in our end [01:13:08] tgr: not a huge problem but a problem [01:13:30] is the log file on vanadium? [01:13:48] is it too big? [01:13:56] and getting truncated? [01:14:19] easy way to check: see if there's a ';' at the end of the URL [01:14:32] the payload is not particularly large [01:15:21] the logs are truncated at 1024 bytes, but there must be many other events larger than that so that's probably a logging issue [01:22:19] Analytics-EventLogging, Analytics-Kanban: Some events not validating for MultimediaViewerNetworkPerformance - https://phabricator.wikimedia.org/T91347#1080700 (Nuria) I "suspect" this is URL decoding not working as well as it should. [03:56:15] o/ milimetric [07:02:14] Analytics, Language-Engineering, Blocked-on-Analytics: Updated languages are not appearing on Language Dashboard - https://phabricator.wikimedia.org/T91369#1081203 (KartikMistry) NEW [07:02:51] Analytics, Language-Engineering, Blocked-on-Analytics, LE-Sprint-83: Updated languages are not appearing on Language Dashboard - https://phabricator.wikimedia.org/T91369#1081211 (KartikMistry) [07:07:38] Need Dan for ^^ :) [07:47:35] Analytics-EventLogging, Popups: Large number of popup events not validating - https://phabricator.wikimedia.org/T91272#1081255 (Prtksxna) [15:24:58] nuria: the /report/results path thing can be explained by my comments in the previous version: [15:24:58] https://gerrit.wikimedia.org/r/#/c/181770/4..6/wikimetrics/controllers/reports.py [15:25:27] I'm not sure if the fix is good, but it looked ok when I quickly looked [15:32:16] milimetric: looking [15:33:21] milimetric: let me try to repro [15:33:38] nuria: don't we have staff now? [15:33:49] at 45, right? [15:33:58] oh, sorry, didn't know [15:47:59] hi madhuvishy! [15:50:13] (CR) Nuria: Add user names to json report, corresponding tests (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/181770 (https://phabricator.wikimedia.org/T74747) (owner: Fhocutt) [15:50:16] (PS7) Nuria: Add user names to json report, corresponding tests [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/181770 (https://phabricator.wikimedia.org/T74747) (owner: Fhocutt) [15:50:43] milimetric: just checked wikimetrics change, makes sense. [15:50:59] (CR) jenkins-bot: [V: -1] Add user names to json report, corresponding tests [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/181770 (https://phabricator.wikimedia.org/T74747) (owner: Fhocutt) [16:19:19] mforns: there are a bunch of alarms coming in about not validated events [16:19:38] nuria, yes, and this time is more serius it seems [16:19:52] mforns: ok let me take a look [16:19:54] I was looking at graphite [16:26:47] nuria, it doesn't seem to be a specific event that causes the alerts, all events have like 5 times the normal error count [16:38:01] nuria, how did you paste yesterday that graphite graph to me? was it just an image? [16:38:27] you can right click on image and it will tell you rendering url [16:38:53] oh, thanks [16:38:54] http://graphite.wikimedia.org/render/?width=1203&height=574&_salt=1425399751.135&target=divideSeries(eventlogging.overall.valid.count%2Ceventlogging.overall.raw.count)&from=00%3A00_20150228&until=23%3A59_20150303 [16:39:38] this is the valid rate, always close to 1 (even with yesterday's alerts) [16:39:52] but today it went down [16:40:22] also, there is a hole in the count of logs (valid and invalid) [16:40:40] mforns: ya, sounds like statsd problem [16:40:57] mforns: this are the counts of not validating events before logs rotated: [16:41:00] https://www.irccloud.com/pastebin/vzJR5j1J [16:43:42] nuria: are you ready to talk @showcase about your estimate on users without JS support? [16:43:59] kevinator: sure [16:44:02] nuria: do you want to talk about backfilling? I can remove that section if you don’t [16:44:17] kevinator: no, that is kind of not so cool [16:44:30] ok, I’ll remove that page [16:46:45] mforns: is there anything you want to showcase (I don’t think so, but want to check)? [16:47:19] I don't think so kevinator, I've been working these last 2 weeks in the scheduler, but it still not showcaseable [16:47:35] thanks for asking :] [16:47:43] mforns: :-) [16:56:39] mforns: there is also a bunch of events related to guided tour, do you wnat to open a ticket for those to the team? [16:56:43] https://www.irccloud.com/pastebin/LOxHobOn [16:56:50] ok [16:57:21] mforns: Please include info about testing on beta labs, link is available in wikitech [16:57:32] sure [16:57:41] mforns: did you find anything else besides these? [16:57:54] no... [16:58:06] did you look at client-side and server-side? [16:59:32] mforns: no, i did not looked at server-side, FYI server side events are 30% of inflow so they have less influence on alarms [16:59:43] ok [17:00:42] nuria, I have an appointment, will have to leave in 5 minutes [17:00:54] mforns: ok, will write ticket [17:01:58] do you think we are losing all events? or just one schema? [17:03:51] mforns: if they are invalid we are not loossing them, they just do not work as sent [17:04:10] mforns: ah you mean in the overall flow? [17:04:20] mforns: I doubt we might, traffic is pretty low [17:04:27] ok [17:04:40] milimetric: what was matt flashen's new team name? [17:04:42] because, If it is something critical, I can stay to work on that [17:04:55] nuria, ^ [17:05:03] it's "Collaboration" [17:05:17] he's in #wikimedia-collaboration [17:07:42] mforns: no need to stay [17:07:50] milimetric: k [17:09:24] nuria, ok, see you all in 2 hours [18:10:49] milimetric: nuria, for string fields in event logging do you know if there's a defined upper limit on length? we have an unstructured string field in a schema, and i'm wondering if it's mandatory to truncate it. i think in practice we won't actually have to do that unless there's a known limit (in that case i wonder if it would make sense for the system to do autotruncation as well to avoid lost messages). this is _not_ related to the [18:10:49] saved pages schema :) [18:16:57] dr0ptp4kt: Our concern is more to store data that we are going to be able to analyze, for example free form text is hard to analyze. What is teh use case ? [18:17:53] nuria: it's the field with a key of "text" in https://meta.wikimedia.org/w/index.php?title=Schema:MobileWikiAppShareAFact [18:18:31] dr0ptp4kt: and what questions do you expect to answer by querying that data? [18:21:12] nuria: i'm not doing the data analysis on it, but i believe the gist is to infer the practical lower and upper bound on share text that results in sharing. to an unknown extent, it could also help in knowing why perhaps sharing fails for a variety of reasons (e.g., overlong text, byte sequences invalid on the sharing target, etc.). [18:21:55] dr0ptp4kt: If you want to infer length then i will store length not text (saving some user bandwith and ours) [18:22:32] nuria: overlong text is the easy case, it's the other cases that are harder [18:22:48] nuria: all of this said, is there an official "limit" on string length? [18:23:09] dr0ptp4kt: yes, there are, url length (depends on proxies & browser) and db size [18:23:43] nuria: okay, so on the wmf infrastructure side it's more a function of the database schema for a given table, right? nothing else? [18:24:01] dr0ptp4kt:and performance right? [18:24:10] nuria: doubtless :) [18:24:12] dr0ptp4kt: we do not wnat to be parsing huge blob of texts [18:24:52] dr0ptp4kt: so if you can limit it to say, arbitrary length of [18:26:02] 100 chars that would be best [18:26:16] so it fits on a regular varchar field [18:26:25] nuria: k, i'll check on that...thanks. sorry, not trying to be a total pita :) [18:26:46] I think text gets mapped to varchar(191)...? [18:26:54] dr0ptp4kt^ [18:27:35] dr0ptp4kt: no pita at all , we like users that ask questions... [19:20:59] yo ori, yt, i got a statsv question for ya [19:26:01] mforns: you are going to follow up in the mediaviewer event issues , right? [19:26:35] nuria, yes [19:26:39] mforns: k [19:27:27] I forgot my cellphone in the car :/... brb in 5 mins [19:38:38] thanks for the comments, nuria! [19:38:53] fhocutt: np, thanks for your work [20:39:12] https://gerrit.wikimedia.org/r/#/c/193985/ review would be appreciated [20:58:35] hi [20:58:58] oops, wrong window :-( [21:06:37] Ironholds: i leave ottomata to merge your change [21:06:54] Ironholds: sorry , " i leave the merge up to ottomata " [21:06:56] nuria, yep, I saw [21:07:00] thanks :) [21:27:37] milimetric, ottomata, how goes hacking? [21:29:44] pretty coooooOOOl [21:29:52] ottomata: what was the question? [21:29:53] realtime pageview counts stream kinda working! [21:30:01] based on your pageview implementation! [21:30:25] dan is gonna help me make it look pretty, am now seeing if I can do top 10 trending page titles too [21:30:26] dunno esy [21:30:28] yet [21:31:41] ottomata, sweet! anyone got free time for a labs/ports/server question? I got the web proxy working but there's some kind of port disjoint [21:31:47] I figured y'all know more about that than I do [21:32:14] ori, give me a few minutes, i am in the thick of it... :) [21:32:18] sorry [21:32:24] ha, i mean, Ironholds (or i sorry for ping) [21:32:39] sure! [21:32:47] just LMK when would be best for me to batcave :) [21:46:02] (PS1) Joal: Add mobile monthly uniques job in oozie. [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 [21:57:43] ok, Ironholds, i'm stuck with my thing, lemme help you for a bit [21:57:45] heading to batcave [21:57:52] cool [21:57:58] I'm just finishing up a thing for legal [22:05:24] tgr, hi [22:07:51] hi mforns [22:09:40] one question please [22:09:51] Ironholds: https://gist.github.com/ottomata/4e0193bb8b2f46f8b861 [22:10:10] how can I generate in the browser a MultimediaViewerNetworkPerformance event with type='image'? [22:10:21] I mean, when is it generated? [22:12:08] on every image view, if $wgNetworkPerformanceSamplingFactor is set [22:12:12] mforns: ^ [22:12:32] oh ok [22:12:37] thanks tgr [22:28:19] kevinator: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/QueryUsingUDF#Sampling_Data:_Get_a_user_agent_report_for_the_past_month [22:28:27] https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/QueryUsingUDF [22:57:53] Analytics, operations: investigate txstatsd error logs - https://phabricator.wikimedia.org/T91464#1084838 (BBlack) NEW a:fgiunchedi [23:16:05] nuria, read your message, what do you mean with the raw stream? [23:31:51] Analytics-EventLogging, Analytics-Kanban: Some events not validating for MultimediaViewerNetworkPerformance - https://phabricator.wikimedia.org/T91347#1085018 (mforns) I managed to reproduce the error from the browser. Setting this beforehand in the browser's console disables sampling (Thanks Gergo): mw.... [23:33:13] Analytics-EventLogging, Analytics-Kanban: Some events not validating for MultimediaViewerNetworkPerformance - https://phabricator.wikimedia.org/T91347#1085023 (mforns) Sorry, I forgot the examples: Here is the url from the browser: ``` https://bits.wikimedia.org/event.gif?%7B%22event%22%3A%7B%22type%22%...