[04:34:47] (03PS1) 10Nuria: Release of analytics.wikimedia.org [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/352075 [04:37:20] (03CR) 10Nuria: [C: 04-1] "Let's not merge this until issues with dashiki extension are sorted out and we can edit the reportcard configuration. cc @milimetric" [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/352075 (owner: 10Nuria) [04:40:19] 06Analytics-Kanban: Global Unique Devices Counts - https://phabricator.wikimedia.org/T143927#3237647 (10Nuria) [04:42:04] 06Analytics-Kanban: Measuring non pageview requests - https://phabricator.wikimedia.org/T162310#3237651 (10Nuria) 05Open>03Resolved a:03Nuria Duplicate of T164019 [04:42:52] 10Analytics: Measure portal pageviews (wikimedia.org) - https://phabricator.wikimedia.org/T162618#3237659 (10Nuria) [04:42:54] 06Analytics-Kanban: Webrequest tagging and distribution. Measuring non-pageview requests - https://phabricator.wikimedia.org/T164019#3218528 (10Nuria) [04:43:07] 10Analytics: Measure portal and hovercard pageviews (wikimedia.org) - https://phabricator.wikimedia.org/T162618#3168814 (10Nuria) [04:43:19] 10Analytics: Measure portal and hovercard pageviews - https://phabricator.wikimedia.org/T162618#3168814 (10Nuria) [04:46:27] 10Analytics, 10Mobile-Content-Service, 06Reading-Infrastructure-Team-Backlog, 06Wikipedia-iOS-App-Backlog: As an end-user I shouldn't see non-articles in the list of trending articles - https://phabricator.wikimedia.org/T124082#1945028 (10Nuria) In order to do this we need the page id, and the computation... [07:02:21] o/ [07:02:30] https://grafana.wikimedia.org/dashboard/db/piwik?orgId=1&from=now-24h&to=now [07:02:44] proved that what is killing piwik is the archiver cronjob :( [07:03:02] now we'll pay a big price only every day at 8 UTC [07:03:19] let's see how heavy it will be [08:03:15] * elukey afk for a bit [08:50:31] (03PS1) 10Joal: Correct daily last access uniques druid laoding [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352091 [08:50:47] (03CR) 10Joal: [V: 032 C: 032] "Self merging typo." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352091 (owner: 10Joal) [09:02:24] joal: o/ [09:02:39] \o [09:02:53] do you mind to give me some info about our usage of sqoop? [09:03:13] my knowledge is a bit outdated and incomplete :) [09:03:48] elukey: Production-wise, we only have labs-to-hdfs monthly job [09:04:18] does it store stuff directly in parquet? [09:04:22] This job loops through all-projects and a subset of tables, and launch 1 sqoop job for each (project, table) [09:05:12] elukey: This represents ~ 750 project * 8 tables (as of today, I expect number of tables to grow) [09:06:11] so sqoop runs as oozie job IIUC [09:06:58] reads from our the labs mariadb slave and stores data on hdfs (parquet format?) and then we run our jobs for mw history on top [09:07:05] elukey: nope, cron job - sqoop is the initial data provider (as camus is) - better to have it not from oozie [09:07:27] elukey: actually not better for real, but easier [09:07:44] ahhhh okok [09:07:53] makes sense? [09:07:57] and where it runs? An1003? [09:08:25] I am trying to get where we should update the mysql-connector [09:09:10] elukey: I'd say every cluster node, +1003, and possibly stat100[234] [09:09:28] auto-answer - on all the workers since like camus creates map-reduce jobs [09:09:32] elukey: finger in the air guess though :) [09:10:55] joal: shall we give the green light to Moritz to update the connector? [09:11:09] then we'll carefully test it during the next days [09:11:17] I have no other idea about how to properly test it [09:11:31] elukey: I don't know either [09:11:58] the other use case is direct access to the hive metastore db, but if anything comes up we'll see job failures [09:12:02] with clear messages [09:12:14] elukey: I think regular hive usage will first tell us if anythning is wrong (hiveServer <-> MySql connections are very regular) [09:12:24] yeah [09:12:55] moritzm: let's upgrade the mysql-connector on Monday ok? [09:13:03] so we will not have a fallout over the weekend :) [09:13:27] you gave me the green light yesterday? it's all upgraded already [09:14:23] huhuhu :) [09:14:34] elukey: looks like it works (for hive at least ;) [09:14:59] moritzm: I wanted to test sqoop yesterday but didn't find time, will do it this morning. [09:26:12] moritzm: ahahhaha sorry I was convinced that we were still on hold, sorry PEBCAK [09:26:55] ah yes I completely lost the "ok doing it now" message [09:27:01] thanks :) [09:31:43] if anything breaks, let me know and I can downgrade, but that's very unlikely to be needed [09:32:08] yep yep [09:32:12] sorry for the noise [09:38:06] 06Analytics-Kanban, 13Patch-For-Review, 15User-Elukey: Metrics and Dashboards for Piwik - https://phabricator.wikimedia.org/T163204#3238008 (10elukey) The Dashboard now looks good and we have enough information to check Piwik's behavior over time. The current version only shows Piwik's effect on basic host m... [09:39:13] 06Analytics-Kanban, 15User-Elukey: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#2990326 (10elukey) a:03elukey [09:39:27] (03PS1) 10Joal: Add uniques global jobs and correct uniques [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352099 (https://phabricator.wikimedia.org/T143928) [09:50:38] joal: are you familiar with naming convention for EL's tables? [09:50:46] otherwise I'll ask to Master Marcel [09:51:09] elukey: I think it's something aroud schema-name + schema-revision [09:51:22] what about WikimediaBlogVisit_5308166_15423246 ? [09:51:22] :D [09:51:45] maybe major/minor versions? [09:51:47] (really?) [09:52:27] no _15423246 seems present everywhere in the double schema-revision [09:52:51] elukey: capsule revision? [09:56:11] that might be an explanation [09:56:30] checking [09:57:06] joal: you are indeed right sir :) [09:57:42] this was probably part of the manuver to move from old to new ua capsule [09:57:50] elukey: I think s [09:59:00] (03PS2) 10Joal: Add uniques global jobs and correct uniques [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352099 (https://phabricator.wikimedia.org/T143928) [09:59:52] (03CR) 10Joal: "dryrun tested - everything seem fine." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352099 (https://phabricator.wikimedia.org/T143928) (owner: 10Joal) [10:17:50] 06Analytics-Kanban, 06DC-Ops, 06Operations, 10ops-eqiad: analytics1030 stuck in console while booting - https://phabricator.wikimedia.org/T162046#3238143 (10elukey) @Cmjohnson any news on an1030? :) [10:22:45] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 10ops-eqiad, 15User-Elukey: Analytics hosts showed high temperature alarms - https://phabricator.wikimedia.org/T132256#3238164 (10elukey) Tried again today: ``` ===== NODE GROUP ===== (1) analytics1060.eqiad.wmnet ----- OUTPUT of 'grep "Hardware e..... [11:06:12] * fdans lunch! [11:20:11] * elukey lunch! [11:52:50] taking abreak a-team [12:02:25] !log removed /etc/cron.d/piwik-archive on bohrium, now puppet creates it for user www-data [12:02:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:11:08] 10Analytics-Tech-community-metrics, 10Phabricator: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#3238369 (10Aklapper) Took another look at this due to hashar's last comment. Regarding Grimoire, [[ https://github.com/grimoirelab/GrimoireELK/blob/master/g... [12:20:24] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 10ops-eqiad, 15User-Elukey: Analytics hosts showed high temperature alarms - https://phabricator.wikimedia.org/T132256#3238415 (10elukey) This is also interesting: ``` ===== NODE GROUP ===== (1) kafka1018.eqiad.wmnet ----- OUTPUT of 'grep "Hardware... [12:54:37] hey a-team ;] [12:56:43] mforns: o/ [12:56:56] sorry I was already afk yesterday, didn't see your ping :( [12:57:03] hey elukey was reading your email [12:58:08] np, I have to delete a data set from druid/pivot, and joseph told me that after deleting it I should ask you to restart druid [12:59:50] sure! [13:00:00] k, will do it now [13:00:12] not sure why we need to restart druid though [13:00:18] maybe only pivot? [13:02:36] elukey, sorry, pivot [13:02:42] ahhh okok :) [13:03:28] mforns: about the two postfix! I wondered the same thing this morning but Joseph solved the mistery - it was before the new UA capsule [13:03:45] oh ok [13:03:50] I think it was the trick used to swap the tables [13:04:00] I see [13:06:45] elukey, I wiped the banner_activity_minutely_sanitization_test data set, can you please restart? :] [13:07:36] yessir [13:08:05] !log restart Pivot on thorium after banner_activity_minutely_sanitization_test cleanup [13:08:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:08:15] done! [13:09:09] elukey, wow, why isn't the pageviews hourly data set there in pivot? [13:10:27] elukey, I can see it in the coordinator administration UI, but not in pivot... [13:11:15] I was freaking out for a sec, thought I had screwed [13:13:59] May 5 13:12:39 thorium pivot[17209]: Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception [13:14:36] O.o [13:14:59] seems started today at ~13:05 UTC after Got the latest time for 'pageviews-hourly' (2017-05-05T13:04:00.000Z) [13:15:02] mmmmmm [13:17:28] https://github.com/implydata/pivot/issues/28 [13:17:37] elukey, it coincides with the moment I wiped banner_activity_test [13:17:48] no data for 2 weeks == bug [13:18:15] but [13:18:21] it also says 'Cluster 'druid' has never seen 'pageviews-hourly' and will introspect 'pageviews-hourly' [13:18:28] then Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception [13:19:03] elukey, before deleting banner_data I could see the data set in druid menu [13:20:20] how do you check that? [13:20:22] (never done it) [13:20:42] batcave for couple mins? [13:21:57] gotta run an errand, be back before standup [13:22:27] k! [13:22:35] mforns: sure! [13:44:49] wow coool [13:44:50] https://github.com/ipfs/blog/blob/uncensorable-wikipedia/src/24-uncensorable-wikipedia/index.md [13:47:53] joal, yt? [13:48:51] ottomata: Cluster 'druid' could not introspect 'pageviews-hourly' because: null exception - this is from pivot [13:49:02] ? [13:49:08] haha [13:49:09] ok [13:49:39] i am not up to date on what datasources should be in druid/pivot [13:49:44] pageview-hourly is one? [13:50:52] I think there is something strange in that data source now, we just discovered it after a pivot restart [13:51:06] trying to get when it started [13:58:18] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Author names that include commata or "and" are split into separate identities in the frontend - https://phabricator.wikimedia.org/T161241#3238765 (10Aklapper) 05Open>03Resolved After testing a bit, confirming this is fixed. Thanks a... [14:02:16] Hey guys, just bakread on pivot/druid [14:03:05] 10Analytics: Pageview hourly data in Pivot is not showing up correctly - https://phabricator.wikimedia.org/T164586#3238782 (10elukey) [14:04:04] seems to be problematic only in pivot: druid has the dataset (as far as it shows on coordinator UI) [14:05:04] joal: opened a task --^ with timings [14:05:17] can you join us in the cave? [14:05:35] yes [14:24:33] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3238850 (10Aklapper) [14:24:41] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3020057 (10Aklapper) @Albertinisg: Yay, thanks. I'm going to do i... [14:24:55] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3238856 (10Aklapper) [14:24:57] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): https://wikimedia.biterg.io shows 2017 contributors who are not listed in mediawiki-identities/wikimedia-affiliations.json - https://phabricator.wikimedia.org/T161235#3238858 (10Aklapper) [14:25:07] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): https://wikimedia.biterg.io shows 2017 contributors who are not listed in mediawiki-identities/wikimedia-affiliations.json - https://phabricator.wikimedia.org/T161235#3125956 (10Aklapper) Merged into T157898 as the underlying problem is... [14:27:57] 06Analytics-Kanban, 06DC-Ops, 06Operations, 10ops-eqiad: analytics1030 stuck in console while booting - https://phabricator.wikimedia.org/T162046#3238863 (10Cmjohnson) an1030's idrac fails to initialize, attempted reboot, drained flea power and still does not initialize. This most likely will require a n... [14:28:43] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017), 07Regression: Only display organizations defined in Wikimedia's DB (disable assuming orgs via hostnames in email addresses) - https://phabricator.wikimedia.org/T161308#3238869 (10Aklapper) [14:28:45] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3238870 (10Aklapper) [14:31:54] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): On the "Git" dashboard, filtering on one organization still lists authors who are with another organization - https://phabricator.wikimedia.org/T157709#3238884 (10Aklapper) [14:31:56] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017), 07Regression: Only display organizations defined in Wikimedia's DB (disable assuming orgs via hostnames in email addresses) - https://phabricator.wikimedia.org/T161308#3238881 (10Aklapper) 05Open>03Resolved @Albertinisg: Yes, this... [14:38:59] ottomata: Not nice finding about druid: looks like reat-time-indexation tasks can't be killed [14:39:45] bwa? [14:40:12] yup [14:41:29] howso? [14:41:32] like, you stop tranquility [14:41:36] and there are indexing tasks, sure [14:41:37] ottomata: I did [14:41:38] but [14:41:43] there's no new data coming in, right? [14:41:48] so, shouldn't they eventually just finish? [14:42:04] ottomata: I stopped tranquility - no data flowing in [14:42:06] https://groups.google.com/forum/#!topic/druid-user/TnyNqh-9g7g [14:42:17] (03PS1) 10Mforns: Deploy vital signs dashboard [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/352159 (https://phabricator.wikimedia.org/T75331) [14:43:49] haha "Scary HTTP status returned" [14:44:12] joal: that's what you are seeing? [14:44:20] a 405 yes [14:44:36] When I send a /shutdown to the task url [14:45:00] huh. [14:45:12] but, the taks isn't doing anything, right? it doesnt' have any new data? [14:45:19] ottomata: Hopefully those tasks will finish [14:45:30] ottomata: I think it lasts an hour [14:45:41] ottomata: So I'm waiting for the moment [14:45:54] (03CR) 10Mforns: [V: 032 C: 032] "Self-merging for deployment" [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/352159 (https://phabricator.wikimedia.org/T75331) (owner: 10Mforns) [14:46:24] aye [14:53:35] milimetric, elukey : piwik no longer is listing visists for financial report [14:53:51] milimetric, elukey : there is only 4 websites listed [14:54:02] 06Analytics-Kanban: Create purging script for mediawiki-history data - https://phabricator.wikimedia.org/T162034#3238975 (10mforns) a:03mforns [14:54:37] milimetric, elukey : or wait, maybe i no longer have permits as administrator? [14:56:13] nuria_: if I log as admin I can see multiple websites, meanwhile if I just log with the apache ldap basic auth I see only 4 (no piwik login though) [14:57:18] elukey: ok i see, i think is no problem, it was just confusing at 1st [14:57:27] super [14:59:43] yep, I went through the same confusion at some point too [15:00:49] ottomata, mforns : standduppp [15:01:38] going! [15:02:52] man it sneaks up so fast [15:03:07] 10Analytics-Tech-community-metrics, 06Developer-Relations (Apr-Jun 2017): On the "Git" dashboard, filtering on one organization still lists authors who are with another organization - https://phabricator.wikimedia.org/T157709#3239019 (10Aklapper) >>! In T157709#3235676, @Albertinisg wrote: > @Aklapper , after... [15:05:18] 06Analytics-Kanban, 13Patch-For-Review: Getting different versions of the same file - https://phabricator.wikimedia.org/T163338#3194051 (10Milimetric) a:05Milimetric>03elukey [15:05:56] 06Analytics-Kanban, 13Patch-For-Review: Getting different versions of the same file - https://phabricator.wikimedia.org/T163338#3194051 (10Milimetric) @Amire80 this change means we're no longer caching for 1 day, so you'll be able to see changes to files you upload more quickly. Let me know if you have questi... [15:22:26] (03Restored) 10Ottomata: Add oozie util workflow to launch spark jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) (owner: 10Ottomata) [15:22:43] (03PS4) 10Ottomata: Add oozie util workflow to launch spark jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) [15:23:33] (03PS5) 10Ottomata: Add oozie util workflow to launch spark jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) [15:26:38] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Make oozie work with spark jobs - https://phabricator.wikimedia.org/T94596#3239132 (10Ottomata) 05declined>03Open [15:26:40] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Mobile PMs has reports on session-related metrics from Wikipedia Apps {hawk} - https://phabricator.wikimedia.org/T86535#3239135 (10Ottomata) [15:27:11] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Make oozie work with spark jobs that use HiveContext - https://phabricator.wikimedia.org/T94596#1167752 (10Ottomata) [15:32:52] joal: , i pushed ^ for review too, if you get around to reviewing stuff [15:32:59] https://gerrit.wikimedia.org/r/#/c/201009/ [15:33:40] 10Analytics-Tech-community-metrics, 10Gerrit: Numerous Gerrit (draft) patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3239157 (10Paladox) @hashar or @Aklapper did it produce an error in the logs? I've back ported this https://gerrit... [15:46:00] 06Analytics-Kanban: Provide unqiues estimate/offset breakdowns externally - https://phabricator.wikimedia.org/T164593#3239203 (10Nuria) [15:46:22] a-team: ssh -N thorium.eqiad.wmnet -L 9089:thorium.eqiad.wmnet:9089 [15:46:26] http://localhost:9089/ [15:46:29] swiv ^ [15:47:29] joal: wikistats meeting? [15:47:45] yes sorry [15:47:50] joal: batcave [15:51:06] (03PS20) 10Ottomata: Spark + JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [15:52:24] (03PS21) 10Ottomata: Spark + JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [16:09:08] hahaha https://phabricator.wikimedia.org/T163483#3237366 [16:09:13] nice neilpquinn [16:09:38] going afk people! [16:09:42] have a good weekend! [16:12:32] 06Analytics-Kanban: Provide uniques offset/underestimate in AQS - https://phabricator.wikimedia.org/T164596#3239290 (10Nuria) [16:12:43] 10Analytics: Provide uniques offset/underestimate in AQS - https://phabricator.wikimedia.org/T164596#3239290 (10Nuria) [16:14:14] 10Analytics: Provide uniques/offset breakdowns available in external unqiue devices files - https://phabricator.wikimedia.org/T164597#3239315 (10Nuria) [16:14:36] 10Analytics: Provide uniques estimate/offset breakdowns available externally - https://phabricator.wikimedia.org/T164597#3239315 (10Nuria) [16:18:58] (03CR) 10Nuria: Add uniques global jobs and correct uniques (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352099 (https://phabricator.wikimedia.org/T143928) (owner: 10Joal) [16:31:04] mforns: I think "monthly pageviews and "pageviews" can probably share annotation file right? [16:31:16] nuria_, yes I agree [16:31:26] mforns: ok, will be changing config [16:36:45] mforns: changed now: https://analytics.wikimedia.org/dashboards/vital-signs/#projects=eswiki,itwiki,enwiki,jawiki,dewiki,ruwiki,frwiki/metrics=MonthlyPageviews [16:37:25] nuria_, awesome thx! [16:55:58] 10Analytics: Provide uniques offset/underestimate breakdowns in AQS - https://phabricator.wikimedia.org/T164596#3239466 (10Nuria) [17:14:32] 06Analytics-Kanban: enforce policy for each Schema [8 pts] {tick} - https://phabricator.wikimedia.org/T102518#3239565 (10dr0ptp4kt) [17:22:06] nuria_: The reason I provided the 2 new fields in current patch is because I added them to the global ones and wanted to be coherent [17:27:09] joal: right, on the new global uniques fiels you mean [17:27:12] correct? [17:27:55] nuria_: correct, I created the new global uniques having the 3 values in the export, and therefore thought for homogenity to include them in the old export as well [17:28:29] nuria_: There were corrections for naming-homogenity in the per-host jobs in any case, so I thought it was ok to include it [17:29:11] joal:ya, those i saw. we will need to revist the split as we need to add it to loading jobs and cassandra too, but we can do that at a later time. [17:29:31] nuria_: I agree, cassandra is a different matter [17:29:51] nuria_: however having them in the archived files makes sense as of this patch I think - Still not ok? [17:30:07] 10Analytics-Tech-community-metrics, 10Gerrit: Numerous Gerrit (draft) patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3239632 (10Aklapper) @Paladox: What is "it"? Which "logs"? And what's the underlying reason to ask that question?... [17:30:56] joal: It is not a big deal, really but if we can split that change in two i think it will be best once for globals and one for file renames and additions of older jobs [17:31:19] k nuria_ [17:34:49] (03PS1) 10Joal: Add last access uniques global oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352181 (https://phabricator.wikimedia.org/T143928) [17:41:10] 06Analytics-Kanban: Update per-hosts-uniques oozie job to match new global ones - https://phabricator.wikimedia.org/T164607#3239658 (10JAllemandou) [17:41:17] (03PS1) 10Joal: Update per host last access uniques oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352182 (https://phabricator.wikimedia.org/T164607) [17:41:39] nuria_: split :) [17:58:51] 10Analytics-Tech-community-metrics, 10Gerrit: Numerous Gerrit (draft) patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3239744 (10Paladox) >>! In T161207#3239632, @Aklapper wrote: > @Paladox: What is "it"? Which "logs"? And what's th... [18:29:56] joal: , yt? [18:30:01] yes [18:30:09] this is working! :) [18:30:13] wondering if you have advice here though [18:30:21] so i have a big list of paths to run the eventlogging refinement on [18:30:26] it'd be nice to run them async [18:30:56] am googling around, I think since I'm just launching them from a single spark master, scala future stuff should work [18:30:56] or [18:31:02] i could turn the list into an RDD [18:31:10] spark as an RDD foreachAsync [18:31:43] ottomata: I've never done it, but I think I prefere the spark one [18:31:51] ok, will try it that way first [18:31:55] ottomata: Congrats on having wotking :) [18:32:07] ottomata: That's reaaly awesome news :) [18:34:45] nuria_: uniques doc updates (https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_Devices and https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_Devices/Last_access_solution) [18:34:50] going for diner ! [18:34:54] Later a-team [18:35:05] bye joal :] [18:36:28] laters! [18:36:32] have a good weekend! [18:43:36] 10Analytics-Tech-community-metrics, 10Gerrit: Numerous Gerrit (draft) patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3239846 (10Aklapper) I have no idea how checking whether an issue gets logged or not would actually bring us close... [19:35:11] * milimetric goin to the airport, will be back later [21:16:02] 10Analytics-Tech-community-metrics, 10Gerrit: Numerous Gerrit (draft) patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3240119 (10hashar) The Gerrit drafts seems to be working all fine now. In T157898#3124564 @Albertinisg mentioned... [22:18:30] (03PS22) 10Ottomata: EventLogging JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [22:19:38] (03CR) 10Ottomata: "Woowee, check out that EventLoggingRefine job. Crazy and it works! Pretty fast too! Still some things to work out, but the general ide" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal)