[01:00:55] (03PS1) 10Catrope: Add frwiki, hewiki, etwiki to list of RCFilters wikis [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/348028 [01:01:08] (03CR) 10Catrope: [C: 032] Add frwiki, hewiki, etwiki to list of RCFilters wikis [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/348028 (owner: 10Catrope) [08:00:15] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Have "Last Attracted Developers" information for Gerrit (already exists for Git) - https://phabricator.wikimedia.org/T151161#3177612 (10jgbarah) >>! In T151161#3176245, @Aklapper wrote: > we are one single boring project. :) This is som... [09:17:37] (03PS3) 10Joal: [WIP] Update banner monthly job to reuse index [analytics/refinery] - 10https://gerrit.wikimedia.org/r/347653 (https://phabricator.wikimedia.org/T159727) [09:29:24] (03PS5) 10Joal: Add oozie job loading daily uniques in druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/347611 (https://phabricator.wikimedia.org/T159471) [09:30:21] (03PS1) 10Joal: Add oozie job loading monthly uniques in druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/348052 (https://phabricator.wikimedia.org/T159471) [13:37:56] 06Analytics-Kanban, 06DC-Ops, 06Operations, 10ops-eqiad: analytics1030 stuck in console while booting - https://phabricator.wikimedia.org/T162046#3178195 (10elukey) Now I am not able to reach the console too. I tried the following without success: ``` elukey@neodymium:~$ sudo ipmitool -I lanplus -H analyt... [14:17:30] whoa joal, what is the source of the monthly data then? [14:17:33] just the data in druid? [14:17:35] minutely? [14:17:38] ottomata: yessir ! [14:21:41] what is minutely? the streaming data? [14:21:56] minutely means we keep minute granularity [14:40:19] ottomata: What do you thing of the hack for the monthly-banners-reindexation? [14:40:39] ottomata: From comments, nuria and mforns were not so happy with it, I tried to explain it better [14:41:50] 10Analytics, 10Analytics-EventLogging, 10MediaWiki-Vagrant, 06Services (watching): Vagrant git-update error for event logging - https://phabricator.wikimedia.org/T161935#3178421 (10Pchelolo) At that point - yes, just tried to rerun it - same thing. I have `vagrant settings nfs_shares off` because of the T1... [14:44:13] joal: i think i understand, but i only briefly read it [14:44:24] instead of re-reading all data from hdfs, you use the more recent data you already ahve in druid [14:44:27] and just aggregate from that? [14:44:33] ottomata: explanations are in the README [14:44:50] the hacky part is waiting a day to do this? [14:44:51] correct, we use already indexed data in Druid [14:45:09] waiting a day assumes that we will def have all of the previous month already in druid [14:45:12] right? [14:46:01] ottomata: Well, we wait a day AND we wait for the previous month being available as well [14:49:29] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 3 others: Switch `/precache` to be a POST end point - https://phabricator.wikimedia.org/T162627#3178459 (10Halfak) [14:54:58] Please give write permission via your FTP software (CHMOD 0755) to your piwik root folder. In some cases, the auto update might still not work, try to change the owner of the piwik to your web server user, or temporarily CHMOD 0777. [14:55:24] * elukey cries [14:55:40] joal: right, but in HDFS, right? not in druid? [14:55:57] ottomata: correct, the assumption is that the daily jobs work [14:56:04] ottomata: I should add that [14:56:11] * joal pads elukey on the back [14:57:24] joal: quick brain bounce before standup on el hive stuff? [14:57:30] ottomata: sure [14:57:50] in bc [15:10:29] 10Analytics-Tech-community-metrics, 07Upstream: When indexing new users, identify identical email addresses and merge identities accordingly in the DB - https://phabricator.wikimedia.org/T151634#3178548 (10Aklapper) This seems to still be an issue and I'd highly welcome investigation: OwlBot in git revision `2... [15:11:23] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): When indexing new users, identify identical email addresses and merge identities accordingly in the DB - https://phabricator.wikimedia.org/T151634#3178553 (10Aklapper) [15:16:30] 10Analytics, 06DC-Ops, 06Operations, 10ops-eqiad, 13Patch-For-Review: Decom/Reclaim analytics1027 - https://phabricator.wikimedia.org/T161597#3178581 (10Nuria) [15:17:46] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Spark + ORES in Hadoop - https://phabricator.wikimedia.org/T162706#3178585 (10Nuria) [15:34:52] 10Analytics-Tech-community-metrics: Maniphest Backend: Provide statistics on resolving tasks - https://phabricator.wikimedia.org/T161926#3178758 (10Aklapper) For the records, **people closing tasks are currently not indexed** in Grimoire. Taking https://phabricator.wikimedia.org/T162636 as an example (a task th... [15:36:11] 10Analytics-Tech-community-metrics: Maniphest Backend: Index which user resolved a task - https://phabricator.wikimedia.org/T161926#3178771 (10Aklapper) [15:39:37] (03CR) 10Nuria: "Does this code need to be parqued until we have the new version of druid?" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/348052 (https://phabricator.wikimedia.org/T159471) (owner: 10Joal) [15:42:29] 06Analytics-Kanban: Measuring non pageview requests - https://phabricator.wikimedia.org/T162310#3178814 (10Nuria) [15:47:16] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 3 others: Switch `/precache` to be a POST end point - https://phabricator.wikimedia.org/T162627#3178852 (10Ottomata) [15:47:47] 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 3 others: Switch `/precache` to be a POST end point - https://phabricator.wikimedia.org/T162627#3178855 (10Ottomata) [15:50:52] 10Analytics: Measure portal pageviews (wikimedia.org) - https://phabricator.wikimedia.org/T162618#3178862 (10Nuria) portal is http://wikimedia.org [15:51:27] 10Analytics: Measure portal pageviews (wikimedia.org) - https://phabricator.wikimedia.org/T162618#3168814 (10Nuria) p:05Triage>03Normal [16:10:59] 10Analytics: Provide cumulative edit count in Data Lake edit data - https://phabricator.wikimedia.org/T161147#3122928 (10Nuria) An approach to do this would be to add a new computation step that computes this values in a smaller denormalized dataset that is not split per year. Once we calculate "edit counts per... [16:22:49] 10Analytics-EventLogging, 06Analytics-Kanban, 10MediaWiki-Vagrant, 06Services (watching): Vagrant git-update error for event logging - https://phabricator.wikimedia.org/T161935#3179183 (10Nuria) [16:30:32] 10Analytics: Secure hue and other private data access sites with 2FA - https://phabricator.wikimedia.org/T159584#3072318 (10Nuria) Let's research whether there is an apache module we can put in front that will handle 2fa [16:31:35] 10Analytics: Secure hue and other private data access sites with 2FA - https://phabricator.wikimedia.org/T159584#3179287 (10Nuria) p:05Triage>03Normal [16:37:01] joal: i need to get lucnh, then want to talk more scala? [16:38:04] sure [17:09:05] 10Analytics, 10Recommendation-API: productionize recommendation vectors - https://phabricator.wikimedia.org/T158973#3179412 (10leila) This task is related to productionizing related-article type and since we have declined that task (T159528), I'll go ahead and decline this as well. Feel free to open it when/if... [17:09:14] 10Analytics, 10Recommendation-API: productionize recommendation vectors - https://phabricator.wikimedia.org/T158973#3179414 (10leila) 05Open>03declined [17:16:10] joal: headed to batcave... [17:16:30] k [17:34:52] * elukey off people! [17:34:53] o/ [18:30:54] joal & milimetric: We're missing you in devops/analytics/research checkin [18:30:55] :P [18:31:57] I realize this is late for joal [18:56:02] 10Analytics, 10Pageviews-API: Endpoint for average view rate in Pageview API - https://phabricator.wikimedia.org/T162933#3180228 (10Halfak) [19:02:57] 10Analytics, 10Pageviews-API: Endpoint for average view rate in Pageview API - https://phabricator.wikimedia.org/T162933#3180228 (10Milimetric) Thoughts on possible implementation: * output table is something like average_monthly_pageviews (page_id, page_title_latest, page_titles_previous, months_included, mo... [19:22:04] (03PS8) 10Ottomata: [WIP] Spark + JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [19:26:48] sorry halfak and others, completely missed the tempo (was dining) [20:19:57] (03CR) 10Milimetric: "I suggest a different cleanup on the private getData method but like the rest." (035 comments) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/347305 (owner: 10Fdans) [20:33:03] 10Analytics, 10Pageviews-API: Endpoint for average view rate in Pageview API - https://phabricator.wikimedia.org/T162933#3180914 (10Halfak) Sounds good @Milimetric @Nettrom was very interested in this during our last discussion. I wrote the SuggestBot use-case based on a conversation with him. [20:53:55] 10Analytics, 06Research-and-Data: geowiki data for Global Innovation Index - https://phabricator.wikimedia.org/T131889#2181884 (10leila) computed data (index, scale) sent to Rafael, plus a description of how the data is generated. [23:40:51] 10Analytics, 06Research-and-Data: geowiki data for Global Innovation Index - https://phabricator.wikimedia.org/T131889#3181864 (10Rafaesrey) Dear Leila, If the results for the latest test are correct, I will not require of an additional data extraction. If Ok with you I will let you know later today as I comp... [23:52:31] 10Analytics, 10Analytics-Cluster: Update SSH fingerprints page for stat1004 - https://phabricator.wikimedia.org/T162972#3181943 (10Tbayer)