[07:03:54] (03CR) 10Elukey: [C: 031] Suppresses subprocess stdout and stderr to avoid false alarms [analytics/refinery] - 10https://gerrit.wikimedia.org/r/445621 (https://phabricator.wikimedia.org/T198966) (owner: 10Fdans) [07:05:29] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10User-Addshore: Investigate June Unique devices increase of 170% for wikidata - https://phabricator.wikimedia.org/T199517 (10Addshore) @Nuria should I file a follow up ticket about adding an annotation to the graph explaining this spike? [08:22:34] (03CR) 10Joal: "Some comments about names, the structure is good :) Thanks fdans !" (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/445395 (https://phabricator.wikimedia.org/T198600) (owner: 10Fdans) [10:05:39] joal: o/ [10:05:59] 10Analytics, 10ChangeProp, 10EventBus, 10WMF-JobQueue, 10Services (designing): Consider disabling automatic topic creation in main-kafka - https://phabricator.wikimedia.org/T199432 (10fgiunchedi) I think a good balance between safety and ease of use would be if kafka could limit the maximum amount of top... [10:06:00] vive la france! :) [10:40:35] (03PS3) 10Sahil505: Changed color shades for sections & charts [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/445574 (https://phabricator.wikimedia.org/T183184) [10:48:53] 10Analytics: Order Data Lake Hardware - https://phabricator.wikimedia.org/T198424 (10elukey) I had a chat with @MoritzMuehlenhoff about this use case, here's some more notes: * there will be no data shared with the Hadoop production cluster or any other host in production. * we (as analytics) will load periodica... [13:44:09] Hi elukey ! [13:44:18] it's been a funny evening yesterday :) [13:44:29] elukey: have you slept enough? [13:44:41] sort of :D [13:44:50] mwarf [14:03:50] 10Analytics: Table view of timely results in wikistats 2 should be ordered in time descending - https://phabricator.wikimedia.org/T199693 (10JAllemandou) [14:21:13] 10Analytics: We should prevent the user from trying to rediscover America - https://phabricator.wikimedia.org/T187452 (10mforns) 05Open>03Resolved a:03mforns Already resolved by Sahil and Amit, thaaanks! [14:30:05] Team- I have an interview now, might be late for standup [14:30:50] ack! [14:49:27] (03CR) 10Sahil505: "https://htmlcolorcodes.com/ is a good source is it feels that color combinations can be further improved :]" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/445574 (https://phabricator.wikimedia.org/T183184) (owner: 10Sahil505) [16:01:37] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10User-Addshore: Investigate June Unique devices increase of 170% for wikidata - https://phabricator.wikimedia.org/T199517 (10Nuria) yes , please, I listed issue on dataset page: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_De... [16:02:39] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10User-Addshore: Investigate June Unique devices increase of 170% for wikidata - https://phabricator.wikimedia.org/T199517 (10Nuria) [16:02:43] 10Analytics, 10Research: [Open question] Improve bot identification at scale - https://phabricator.wikimedia.org/T138207 (10Nuria) [16:02:54] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10User-Addshore: Investigate June Unique devices increase of 170% for wikidata - https://phabricator.wikimedia.org/T199517 (10Nuria) 05Resolved>03stalled [16:02:57] 10Analytics, 10Research: [Open question] Improve bot identification at scale - https://phabricator.wikimedia.org/T138207 (10Nuria) [16:16:24] (03CR) 10Nuria: "+1000 on changes for line chart, things look a lot better." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/445574 (https://phabricator.wikimedia.org/T183184) (owner: 10Sahil505) [16:23:29] 10Analytics, 10WMDE-Analytics-Engineering, 10Wikidata, 10User-Addshore: Investigate June Unique devices increase of 170% for wikidata - https://phabricator.wikimedia.org/T199517 (10Addshore) a:05Addshore>03None [17:56:45] 10Analytics, 10Analytics-Wikistats: Vet calculation of total article count by suming articles over timespam - https://phabricator.wikimedia.org/T199734 (10Nuria) p:05Triage>03High [17:59:22] mforns: yt? [18:01:49] (03CR) 10Nuria: [V: 032 C: 032] Changed color shades for sections & charts [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/445574 (https://phabricator.wikimedia.org/T183184) (owner: 10Sahil505) [18:03:44] 10Analytics, 10Analytics-Wikistats: Wikistats 2. Total article count metric. - https://phabricator.wikimedia.org/T199735 (10Nuria) p:05Triage>03High [18:04:09] joal: yt? [18:13:06] (03PS2) 10Nuria: Suppresses subprocess stdout and stderr to avoid false alarms [analytics/refinery] - 10https://gerrit.wikimedia.org/r/445621 (https://phabricator.wikimedia.org/T198966) (owner: 10Fdans) [18:13:46] (03CR) 10Nuria: [V: 032 C: 032] "Self-merging after update of commit message." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/445621 (https://phabricator.wikimedia.org/T198966) (owner: 10Fdans) [18:14:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Fix sqoop script so that the jar-generation step doesn't print logs (alerts email sent by cron) - https://phabricator.wikimedia.org/T198966 (10Nuria) [18:23:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Varnishkafka eventlogging instances delivery failures - https://phabricator.wikimedia.org/T198070 (10Nuria) 05Open>03Resolved [18:31:37] nuria_, hi missed your ping [18:33:16] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog, 10Readers-Web-Kanbanana-Board: Some VirtualPageView are too long and fail EventLogging processing - https://phabricator.wikimedia.org/T196904 (10Nuria) @phuedx developers with access to stats machines (i think most (all?) of y... [18:33:33] mforns: i think i am missing something with total article count [18:34:39] mforns: do we have now an "article created metric"? [18:34:56] nuria_, I think yes [18:35:19] nuria_, https://stats.wikimedia.org/v2/#/all-projects/contributing/new-pages/normal|bar|2-Year~2016060100~2018071600|~total [18:35:21] new pages no? [18:37:18] mforns: for that to work it needs the splits by namepsace and such to be able to filter "articles" from "pages" [18:38:30] 10Analytics, 10Analytics-Wikistats: Wikistats 2. Total article count metric. - https://phabricator.wikimedia.org/T199735 (10Nuria) [18:39:44] 10Analytics, 10Analytics-Wikistats: Vet calculation of total article count by summing pages created (with proper filters) over timespam - https://phabricator.wikimedia.org/T199734 (10Nuria) [18:44:59] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Vet calculation of total article count by summing pages created (with proper filters) over timespam - https://phabricator.wikimedia.org/T199734 (10Nuria) [18:47:24] nuria_, the new-pages metric is split by content/no-content would that be enough? [18:48:24] mforns: also needs namespace (which it has) and I am not sure what else but joal might know (the wiki page that describes what an article is is crazy long) [18:48:56] nuria_, I don't think aqs has namespace [18:49:17] mforns: mmm i think it does one sec [18:49:24] no, only page-type [18:49:28] content/no-content [18:49:47] https://wikimedia.org/api/rest_v1/metrics/edited-pages/new/all-projects/all-editor-types/non-content/monthly/2016060100/2018071600 [18:49:49] mforns: ah content is namespace zero then [18:49:53] yes [18:50:02] non-content is everything else [18:50:12] mforns: we will also need editor type i think [18:50:36] new-pages does have editor type [18:50:47] so we're good no? [18:51:58] breaking down article count by editor type will be cool, a bit in the line of wikicron's 90/10 metrics [18:52:30] mforns: yayayay [18:53:59] 10Analytics, 10Analytics-Kanban: Virtual pageview refine should not refine data that does not come from wikimedia domains - https://phabricator.wikimedia.org/T197971 (10Nuria) [18:54:51] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0 Remaining reports. - https://phabricator.wikimedia.org/T186121 (10Nuria) [18:55:24] 10Analytics, 10Analytics-Kanban: Ability to salt and hash in hadoop eventlogging sanitization backend - https://phabricator.wikimedia.org/T198426 (10Nuria) a:03mforns [18:55:31] * elukey off! [18:55:50] 10Analytics, 10Analytics-Kanban: Ability to salt and hash in hadoop eventlogging sanitization backend - https://phabricator.wikimedia.org/T198426 (10Nuria) [18:56:28] 10Analytics-Kanban: Private geo wiki data in new analytics stack - https://phabricator.wikimedia.org/T176996 (10Nuria) 05Open>03Resolved [18:56:30] 10Analytics-Kanban: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280 (10Nuria) [18:57:04] 10Analytics-Kanban, 10Analytics-Wikistats: Create Daily & Monthly pageview dump with country data and Visualize on UI - https://phabricator.wikimedia.org/T90759 (10Nuria) 05Open>03Resolved [18:57:06] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256 (10Nuria) [18:57:21] nuria_, I think that we could present the metric page count, rather than article count, and then offer the breakdown possibility to look at articles separately [18:58:05] mforns: mmm.. not so sure, it might be missleading w/o understanding difference between pages and articles [18:58:12] maybe in the dashboard we can show them filtered by article [18:59:23] 10Analytics: Drop mediawiki history old snapshots from druid public cluster - https://phabricator.wikimedia.org/T197889 (10Nuria) a:03fdans [18:59:24] and if you click there, then the breakdown would be open by default [18:59:48] 10Analytics, 10Analytics-EventLogging, 10Performance-Team (Radar): Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207 (10Nuria) p:05High>03Triage [19:00:02] 10Analytics, 10Analytics-EventLogging, 10Performance-Team (Radar): Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207 (10Nuria) a:03Milimetric [19:03:09] 10Analytics, 10Analytics-Wikistats: Wikistats 2.0: "aa.wikipedia.org" exists and has data available, but marked "Invalid" - https://phabricator.wikimedia.org/T187414 (10Nuria) @Krinkle sqooping is different per wiki size, thus it requires a whilelist to manage it. See similar addition: https://gerrit.wikimedi... [19:03:11] 10Analytics, 10Analytics-Wikistats: Wikistats 2.0: "aa.wikipedia.org" exists and has data available, but marked "Invalid" - https://phabricator.wikimedia.org/T187414 (10Nuria) a:03fdans [19:03:36] 10Analytics: Drop mediawiki history old snapshots from druid public cluster - https://phabricator.wikimedia.org/T197889 (10Nuria) p:05High>03Triage [19:04:16] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0: "aa.wikipedia.org" exists and has data available, but marked "Invalid" - https://phabricator.wikimedia.org/T187414 (10Nuria) [19:04:42] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0: "aa.wikipedia.org" exists and has data available, but marked "Invalid" - https://phabricator.wikimedia.org/T187414 (10Nuria) p:05Normal>03High [19:07:57] 10Analytics: turnilo x axis improperly labeled - https://phabricator.wikimedia.org/T197276 (10Nuria) p:05Normal>03Triage [19:08:35] 10Analytics: turnilo x axis improperly labeled - https://phabricator.wikimedia.org/T197276 (10Nuria) Bug on turnilo, looks like the workarround in to change your tz .... [19:10:02] 10Analytics, 10Analytics-Wikistats: Wikistats 2: New Pages split by editor type wrongly claims no anonymous users create pages - https://phabricator.wikimedia.org/T185342 (10Nuria) ping @JAllemandou this issue might be significant for "total article count" [19:11:42] 10Analytics: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059 (10Nuria) a:03fdans [23:28:54] 10Analytics, 10Analytics-Kanban: Problems with external referrals? - https://phabricator.wikimedia.org/T195880 (10Nuria) Per conversation with @JKatzWMF * we do not believe that there is an issue with "correlations" of referrers * it is kind of odd that much of our traffic it is tagged as having no referrer,...