[03:26:02] Analytics-Wikistats, Operations, Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#2208982 (Dzahn) I would recommend to accept the regression as a feature. We actually like it when the wikipedia.org domain just... [03:28:14] Analytics, DNS, Operations, Traffic: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2208983 (MZMcBride) Thank you for the explanations and clarifications here. I really appreciate them. [03:31:38] Analytics-Wikistats, Operations, Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#2208987 (Dzahn) Interestingly right after i say this i see the discussion on T13240 to introduce analytics.wikimedia.org. And i... [03:35:13] Analytics, DNS, Operations, Traffic: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2197243 (Dzahn) also see T126281 (i think we should not fix/redirect stats.wikipedia.org, but say that there is just stats.wikimedia.org and this new analytics.wikimedia.org [04:30:30] Analytics-Wikistats, Operations, Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#2010000 (Nemo_bis) >>! In T126281#2208987, @Dzahn wrote: > Interestingly right after i say this i see the discussion on T132407... [09:21:33] Not sure where to report this, but I’m seeing problems with the latest pagecounts-*totals files in http://dumps.wikimedia.org/other/pagecounts-ez/merged/. The most recent 4 files are quite a lot smaller than the previous ones, and missing about 80% of the en.wikipedia titles I want to look up. E.g. try bzgrep 'Haplochromis_gigas' pagecounts-2016-03-views-ge-5-totals.bz2 [09:22:07] I’ve asked about it at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Missing_pages_in_page_counts_files but no reply, so that’s probably the wrong place. [10:30:07] Analytics, Hovercards, Unplanned-Sprint-Work, Reading-Web-Sprint-70-Lady-and-the-Trumps: Capture hovercards fetches as previews in analytics - https://phabricator.wikimedia.org/T129425#2209371 (phuedx) a:dr0ptp4kt [10:30:18] Analytics, Hovercards, Unplanned-Sprint-Work, Reading-Web-Sprint-70-Lady-and-the-Trumps: Capture hovercards fetches as previews in analytics - https://phabricator.wikimedia.org/T129425#2105269 (phuedx) … per the above. [10:50:21] Analytics-General-or-Unknown, Language-Engineering, Mobile-Apps, Wikipedia-Android-App-Backlog, and 2 others: there should be a comparison of clicks count on interlanguage links on different platforms - https://phabricator.wikimedia.org/T78351#2209433 (Nemo_bis) [12:59:47] yan___: I'll make a phabricator task and cc Erik Zachte who is still maintaining that dataset. He's handing that over to us, so in the future this is a good place to report it. [12:59:58] yan___: do you have a phabricator account I can cc you with? [13:01:25] Analytics: pagecounts-ez files are missing data - https://phabricator.wikimedia.org/T132761#2209683 (Milimetric) [13:02:07] milimetric that’s great, thanks. I think my phabricator username is HYanWong [13:02:26] Analytics: pagecounts-ez files are missing data - https://phabricator.wikimedia.org/T132761#2209697 (Milimetric) @ezachte: this seems like a good opportunity to transition the maintenance of this dataset. I'm going on vacation now for 2 weeks :/ but if you could take some notes while you look at this issue,... [13:02:55] k, yan___ I subscribed you : https://phabricator.wikimedia.org/T132761 [13:03:21] joal: you around? [13:03:33] milimetric thanks - really helpful. [13:03:59] just to set expectations, yan___, if Erik has time I'm sure he'll look at it but he's quite busy these days [13:04:21] I'm going on vacation now for 2 weeks, but when I'm back I'll take a look if nothing's happened [13:04:23] milimetric sure. there’s no urgency on this, but would be nice to get sorted. [13:04:35] yan___: in the meantime, do you know about the pageviews dataset? [13:04:52] milimetric: I know there’s a new API [13:04:58] Hey milimetric [13:05:13] milimetric: but unclear if there are per-month download dumps [13:05:15] Sorry, was python-driven [13:05:38] yan___: right, we're also making available some of the data behind the API. The pagecounts-ez is just a re-processing of the pageviews dataset. You can find out more details here: https://dumps.wikimedia.org/other/analytics/ and the pageviews dataset here: https://dumps.wikimedia.org/other/pageviews [13:06:03] joal: you took like 10 seconds to answer me, unacceptable, let's get that down to 0.1 seconds [13:06:10] :P [13:06:22] ok, I wanted to make sure everything's good while I'm gone [13:06:25] * joal tries to be API style ;) [13:06:37] I think I managed to phase myself out of any work over the last couple of weeks but just in case, is there anything stuck on me? [13:06:49] milimetric: nothing on my side [13:06:57] did you want to finish up looking through wikistats? [13:07:05] or was our initial poke enough for you? [13:07:15] milimetric: I'll move forward with Wikistats dashboards review and try to take notes [13:07:47] I mean I'm happy to work with you on that, I don't really have any other major things to pick up [13:07:47] milimetric: I'd like to spend another bunch of hours, making sure I have a better view of what's at stake :) [13:07:56] milimetric: can’t see any aggregated data in the new files - so for monthly totals I guess I’m best sticking to -ez processed data [13:08:25] milimetric: for sure ! Was just saying I can continue to move on while you're on holidays :) [13:08:37] yan___: yes, those files are at the page and project level only, so you'd have to crunch monthly aggregates yourself. But just in case it becomes more urgent and you need something, those files are there [13:08:59] joal: well, if you want to do it today anytime, let me know [13:09:01] milimetric: cheers fo the heads up. Over and out. [13:09:07] :) [13:09:27] milimetric: hm, we can do an hour or so before standup :) [13:09:38] milimetric: currently pythoning for halfak :) [13:11:15] sure, just offering. btw, have you seen http://techcrunch.com/2016/04/14/kite-wants-to-be-every-developers-pair-programming-buddy/ [13:11:57] milimetric: I hadn't see that ! [13:12:01] sounds just awesome ! [13:13:18] seems nice, and they seem to be focusing on python [13:44:31] Analytics-EventLogging, Analytics-Kanban: EventLogging dies when fetching a schema over HTTP that does not exist. {oryx} - https://phabricator.wikimedia.org/T124799#2209902 (mforns) I tried reproducing the described error and could not: --- **When sending an event with a non-existing revision**, the Me... [13:59:51] Analytics-EventLogging, Operations: "Throughput of EventLogging NavigationTiming events" UNKNOWN - https://phabricator.wikimedia.org/T132770#2209918 (fgiunchedi) [14:02:58] If anyone is awake… I could use guidance about stats.wikipedia.org, https://phabricator.wikimedia.org/T126281. [14:03:29] My preference would be to close that bug and leave it broken, since using .wikipedia.org urls for non-wikis is frowned upon. But I don't want to ruin anyone's ongoing work... [14:25:32] hey good morning! sorry yall, had a PT appt. this morn [14:27:19] milimetric: I decline the invite to review wikistats for today, not enough energy :) [14:29:12] good! +1 for self-preservation :) [14:52:21] Analytics-EventLogging, Analytics-Kanban: EventLogging dies when fetching a schema over HTTP that does not exist. {oryx} - https://phabricator.wikimedia.org/T124799#2210114 (Ottomata) OHHHH it's the MySQL consumer that died. Right, I think this might make more sense. I think this will happen if EventLo... [14:53:18] Analytics-EventLogging, Analytics-Kanban: EventLogging dies when fetching a schema over HTTP that does not exist. {oryx} - https://phabricator.wikimedia.org/T124799#2210117 (Ottomata) If you run a pipe data to a MySQL consumer for which the schema doesn't exist, I betcha it will die. [15:29:35] ah, dashes!!! [15:32:32] Analytics: Unique devices Graphana Dashboard - https://phabricator.wikimedia.org/T132795#2210412 (Nuria) [15:32:43] Analytics: Unique devices Graphana Dashboard - https://phabricator.wikimedia.org/T132795#2210424 (Nuria) p:Triage>High [15:33:08] Analytics-Kanban: Unique devices Graphana Dashboard - https://phabricator.wikimedia.org/T132795#2210412 (Nuria) [15:34:51] Analytics-Kanban: Unique Devices Endpoint: Write documentation - https://phabricator.wikimedia.org/T132794#2210434 (Nuria) [15:35:24] Analytics-Kanban: Document the new AQS Unique devices endpoint and launch it - https://phabricator.wikimedia.org/T129520#2210435 (Nuria) [15:36:19] Analytics-Kanban: Unique Devices Endpoint: Write documentation - https://phabricator.wikimedia.org/T132794#2210400 (Nuria) Open>Resolved [15:36:31] Analytics-Kanban: Document the new AQS Unique devices endpoint and launch it - https://phabricator.wikimedia.org/T129520#2108010 (Nuria) Unique Devices Endpoint: Write documentation Let's write docs about how to query the unique devices data similar to pageview APi. A section with a "quickstart" and example... [15:37:42] Analytics-Kanban: Group flavours of desktop Windows together in OS reports - https://phabricator.wikimedia.org/T131125#2210438 (Nuria) We are following UA parser convention on this regard. Windows 7 and windows 95 have little in common and thus are reported as separate OS. [15:37:52] Analytics-Kanban: Browser reports improvements (parent task) - https://phabricator.wikimedia.org/T130405#2210440 (Nuria) [15:37:54] Analytics-Kanban: Group flavours of desktop Windows together in OS reports - https://phabricator.wikimedia.org/T131125#2210439 (Nuria) Open>declined [15:38:25] Analytics, Commons, Multimedia, Wikidata, and 2 others: Allow tabular datasets on Commons (or some similar central repository) (CSV, TSV, JSON, XML) - https://phabricator.wikimedia.org/T120452#2210441 (Nuria) [15:52:16] Analytics, Analytics-Cluster, Patch-For-Review: Single Kafka partition replica periodically lags - https://phabricator.wikimedia.org/T121407#2210503 (Ottomata) FYI I disabled the MaxLag icinga alert. We should revisit this alert after we upgrade to 0.9. [15:56:11] Analytics-Wikistats, Operations, Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#2010000 (Nuria) stats.wikimedia.org will nor be fully replaced for quite some many months and even then we will keep old "froze... [16:00:43] a-team: standdupppp [16:01:07] sorry ! [16:01:44] coming! [16:02:52] madhuvishy: coming to standup? [16:30:23] madhuvishy: sorry, just saw your e-mail [16:34:18] Analytics-Kanban: Provide browser reports charts as stacked %age of all pageviews - https://phabricator.wikimedia.org/T131126#2210666 (Nuria) Changes to this extent have been deployed, closing. https://browser-reports.wmflabs.org/ [16:34:23] Analytics-Kanban: Browser reports improvements (parent task) - https://phabricator.wikimedia.org/T130405#2210668 (Nuria) [16:34:25] Analytics-Kanban: Provide browser reports charts as stacked %age of all pageviews - https://phabricator.wikimedia.org/T131126#2210667 (Nuria) Open>Resolved [16:35:37] Analytics-Kanban: Browser reports improvements (parent task) - https://phabricator.wikimedia.org/T130405#2135158 (Nuria) [16:35:39] Analytics-Kanban, Patch-For-Review: Add (and default to) a breakdown in percentages also for the line chart. - https://phabricator.wikimedia.org/T130406#2210670 (Nuria) Open>Resolved [16:36:26] Analytics-Kanban: limn-multimedia-data queries raising SQL syntax errors - https://phabricator.wikimedia.org/T132404#2210672 (Nuria) Open>Resolved [16:36:40] Analytics-Kanban, Patch-For-Review: Remove frequency and timeboxed options from limn-x-data repos config files {lama} - https://phabricator.wikimedia.org/T132062#2210673 (Nuria) Open>Resolved [16:37:09] Analytics-Kanban, Release-Engineering-Team: [Spike] Figure out how to automate releases with jenkins {hawk} - https://phabricator.wikimedia.org/T130576#2210674 (Nuria) Open>Resolved [16:37:11] Analytics-Kanban: Get jenkins to automate releases {hawk} - https://phabricator.wikimedia.org/T130122#2210675 (Nuria) [16:38:35] mforns: on your reportupdater docs it says you can get reportupdater from github... but it is on gerrit right? [16:38:38] https://wikitech.wikimedia.org/wiki/Analytics/Reportupdater [16:38:51] nuria_, yes [16:39:00] mforns: k will chnage [16:39:08] nuria should I make this clear? [16:39:13] explicit? [16:39:23] ok, I can change that too :] [16:39:42] mforns: no worries I can change it so as not make people think we accept pull requests via github [16:39:52] I see makes sense [16:41:05] Analytics-Kanban: Update reportupdater's documentation - https://phabricator.wikimedia.org/T132326#2210692 (Nuria) Open>Resolved [16:43:52] Analytics-Kanban, Patch-For-Review: Allow configurable number formatting - https://phabricator.wikimedia.org/T131965#2210699 (Nuria) Open>Resolved [16:44:06] Analytics-Kanban: Cassandra Backfill July {melc} - https://phabricator.wikimedia.org/T119863#2210700 (Nuria) Open>Resolved [16:44:28] Analytics: Visualize unique devices data in dashiki - https://phabricator.wikimedia.org/T122533#2210717 (Nuria) [16:44:30] Analytics-Kanban, Patch-For-Review: Add AQS endpoint for uniques - https://phabricator.wikimedia.org/T129518#2210716 (Nuria) Open>Resolved [16:44:52] Analytics-Kanban, Patch-For-Review: End date not included - https://phabricator.wikimedia.org/T131641#2210718 (Nuria) Open>Resolved [16:46:21] Analytics-Kanban, Patch-For-Review: Pageviews API doesn't work for titles with % symbol. - https://phabricator.wikimedia.org/T131369#2210731 (Nuria) Open>Resolved [16:46:49] Analytics-Kanban, Patch-For-Review: Strip out a www. prefix for the "project" parameter passed into the pageview API - https://phabricator.wikimedia.org/T127030#2210732 (Nuria) Open>Resolved [16:47:18] Analytics-Kanban, Patch-For-Review: Align labels in legend - https://phabricator.wikimedia.org/T131935#2210733 (Nuria) Open>Resolved [16:47:32] Analytics-Kanban, Patch-For-Review: Fix 1-week lag in executing browser reports {lama} - https://phabricator.wikimedia.org/T131049#2210734 (Nuria) Open>Resolved [16:47:50] Analytics-Kanban, Wikipedia-iOS-App-Product-Backlog, iOS-app-Bugs, iOS-app-feature-Analytics, and 2 others: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2210735 (Nuria) Open>Resolved [16:49:42] joal; thanks for keeping up to date the chagelog of pageview definition: https://meta.wikimedia.org/wiki/Research:Page_view#Change_log [16:50:33] Analytics-Kanban, Wikipedia-Android-App-Backlog, Patch-For-Review: Count requests to RESTBase from the Android app - https://phabricator.wikimedia.org/T128612#2210758 (Nuria) Open>Resolved [16:50:45] Analytics-Kanban, Patch-For-Review: Add AQS endpoint for uniques - https://phabricator.wikimedia.org/T129518#2210761 (Nuria) [16:50:47] Analytics-Kanban, Patch-For-Review: Fill new AQS endpoint with data from Hadoop - https://phabricator.wikimedia.org/T129519#2210760 (Nuria) Open>Resolved [16:51:55] Analytics-Kanban: Describe threat model for sanitized pageview data - https://phabricator.wikimedia.org/T131158#2210764 (Nuria) Code here: https://phabricator.wikimedia.org/T118838 [16:52:11] Analytics: Productionize Pageview_sanitization hive code with Oozie job and refinery inclusion {hawk} - https://phabricator.wikimedia.org/T118839#2210766 (Nuria) [16:52:13] Analytics-Kanban, Patch-For-Review: Write hive code doing pageview data anonimisation with two tables {hawk} - https://phabricator.wikimedia.org/T118838#2210765 (Nuria) Open>Resolved [16:52:29] Analytics-Kanban, Patch-For-Review: Fix phab script to gather stats also from point field , not only from title - https://phabricator.wikimedia.org/T130543#2210768 (Nuria) Open>Resolved [16:52:55] Analytics-Kanban: Put quaterly review together for Q3 - https://phabricator.wikimedia.org/T131947#2210769 (Nuria) Open>Resolved [16:53:20] Analytics, Analytics-Cluster, Operations, Traffic: Upgrade analytics-eqiad Kafka cluster to Kafka 0.9 (or 0.10?) - https://phabricator.wikimedia.org/T121562#2210775 (Nuria) [16:53:23] Analytics-Cluster, Analytics-Kanban: Experiment with new Kafka versions and verify that they work with existing clients - https://phabricator.wikimedia.org/T132595#2210774 (Nuria) Open>Resolved [17:13:10] np nuria_ :) [17:15:23] hi a-team, I just tried logging into stat1002 and got a message that the host key changed. Is this normal? [17:15:38] yes [17:15:42] Hi kevinator, it is normal, it's not stat1002 that has changed, but bastion1001:) [17:15:53] ah [17:16:00] just do what it says to clear the entry in known hosts [17:17:49] kevinator: https://discussions.apple.com/thread/1861839?tstart=0 [17:18:57] http://www.gossamer-threads.com/lists/wiki/wikitech/700645 [17:27:49] thanks milimetric joal, i'm in again :-) [17:32:37] bd808: kudos to ya for making your own job [17:33:24] nuria_: :) Now I will have a more difficult time complaining about being asked to work on the wrong things [17:33:55] bd808: jaja totally, but you know any job has that 20% of stuff you just have to live with [17:34:23] *nod* "that's why they call it work" [17:49:18] Analytics-Wikistats, Operations, Patch-For-Review, Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#2210923 (Andrew) Open>declined I've removed the .wikipedia.org puppet code, and I'm closing this... [17:52:51] ottomata: one (probably mega obvious) question [17:54:33] ja? [17:56:42] ottomata: sooo [17:57:04] for our dashiki labs instance: https://horizon.wikimedia.org/project/instances/6111e769-7e1d-4447-a622-59371d8eae62/ [17:57:21] ottomata: how can i know what are teh puppet modules active there? [17:57:31] ottomata: are those configured through horizon? [17:57:50] I don't think so, config is still on wikitech [17:58:01] whoa never logged into horizon before [17:58:20] ottomata: dashiki-01 [17:58:27] what project is that in? [17:58:31] nuria, i usually go here: https://wikitech.wikimedia.org/wiki/Special:NovaInstance [17:58:32] ottomata: horizon needs some bookmarks let me tell you, [17:58:36] and then click on configure [17:58:52] ottomata: but ahem ... i though we did not used that anymore but hey maybe I am wrong [17:59:29] ottomata: my joy to think that nova thing was gone ... [17:59:53] ottomata: k will look [18:03:15] nuria_: we do use it [18:03:20] but not for variables [18:03:23] just for including role classes [18:03:30] variables should be in hiera [18:03:45] a-team have a nice weekend! ciao [18:10:42] laters! [18:24:37] Analytics, DNS, Operations, Traffic: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2211054 (Nuria) @BBlack This would not be a full-fledged service. What we would be deploying either via puppet of fab is just html/js so we only really need an apache install via pup... [18:32:11] Analytics, DNS, Operations, Traffic: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2211069 (BBlack) @Nuria - thanks for the details! We still need to sort out an actual place for the js/html to live at in production (which, if it's as simple as it sounds, can probabl... [19:08:14] Analytics, DNS, Operations, Traffic: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2211146 (Nuria) @BBlack : ganeti sounds fine as really the majority of the time requests are going to be served by varnish. The fabfile we use to deploy to labs is here: https://github... [19:10:50] ottomata: is there a set of hiera files for labs? [19:11:09] sortof [19:11:14] you can commit to puppet [19:11:16] or you can edit in wikitech [19:11:26] wikitech takes presidence [19:11:30] e.g. [19:11:30] https://wikitech.wikimedia.org/wiki/Hiera:Analytics [19:16:55] ottomata: ok, i knew this, but see my brian forgets cause i am in denial about these wikitech pages [19:17:22] *my brain [20:06:07] (PS11) Nuria: Allow filtering of data breakdowns [analytics/dashiki] - https://gerrit.wikimedia.org/r/278395 (https://phabricator.wikimedia.org/T131547) (owner: Jdlrobson) [20:06:49] HaeB: are there any notes about reading's quaterly review? [20:07:35] yes, i guess they will be published soon [20:08:42] milimetric: I have fixed the dashes (i think) but i think there is a raise condition with semantics transition module that is only visible when code is not build [20:09:13] milimetric: will look into that (unrelated to the patch so if an issue it will be another patch) [20:09:38] ...not sure who is handling that though. (in the meantime you can also look at the draft, nonpublic notes - will send you the link) [20:11:08] k, reviewing [20:57:21] (CR) Milimetric: [C: -1] Allow filtering of data breakdowns (5 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/278395 (https://phabricator.wikimedia.org/T131547) (owner: Jdlrobson) [20:57:28] nuria_: the nuria_ wanna talk about this in the cave? [20:57:42] I think I did a terrible job explaining in the review and it's not that bad [21:10:31] i need brainbounce, milimetric, batcave? [21:11:08] omw [21:20:59] (CR) Nuria: Allow filtering of data breakdowns (3 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/278395 (https://phabricator.wikimedia.org/T131547) (owner: Jdlrobson) [21:21:42] milimetric: i think i got it and i have little time until later which will be too late for ya, see if my comments make sense [21:42:05] (CR) Milimetric: Allow filtering of data breakdowns (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/278395 (https://phabricator.wikimedia.org/T131547) (owner: Jdlrobson) [22:38:02] Analytics-Kanban, Wikipedia-iOS-App-Product-Backlog, iOS-app-Bugs, iOS-app-feature-Analytics, and 2 others: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2211500 (Tbayer) To keep things together and spell out more clearly some of the progress on this task this week... [22:55:38] Analytics-Kanban, Wikipedia-iOS-App-Product-Backlog, iOS-app-Bugs, iOS-app-feature-Analytics, and 2 others: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2211513 (Nuria) > you marked this task as resolved. Does this apply to the iOS team's part of the work too? Doe... [23:01:15] Analytics-Kanban, Wikipedia-iOS-App-Product-Backlog, iOS-app-Bugs, iOS-app-feature-Analytics, and 2 others: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2211514 (MBinder_WMF) Just want to reiterate that since this task is pointed, it should have a separate task fo...