[04:48:16] (03PS1) 10Chelsyx: Add mobile_apps_uniques_by_country_daily and mobile_apps_uniques_by_country_monthly jobs in oozie [analytics/refinery] - 10https://gerrit.wikimedia.org/r/451566 (https://phabricator.wikimedia.org/T186828) [04:57:35] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Reading-analysis, 10Patch-For-Review: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828 (10chelsyx) Hi @Nuria, could you review the change https://gerrit.wikimedia.org/r/451566 when you hav... [07:27:00] (03CR) 10Joal: "2 minimal nits, then ok for me once tested :)" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/450861 (https://phabricator.wikimedia.org/T198908) (owner: 10Ottomata) [07:51:46] (03CR) 10Joal: [C: 031] "Looks good to me from an oozie-like perspective (I didn't check validity of the queries) :) Let's wait for somebody else review before mer" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/451566 (https://phabricator.wikimedia.org/T186828) (owner: 10Chelsyx) [10:00:31] 10Analytics, 10MinervaNeue, 10Readers-Web-Backlog, 10Design: [Spike 8hrs] Sticky header instrumentation - https://phabricator.wikimedia.org/T199157 (10ovasileva) [10:36:47] hey teaam :] [11:57:53] (03CR) 10Mforns: [C: 04-1] "Let's not merge this, until saltrotate is productionized, and the salt is passed to ELSanitization job. Otherwise, the job would fail beca" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/450588 (https://phabricator.wikimedia.org/T199902) (owner: 10Mforns) [12:13:22] (03PS1) 10Mforns: Cast to string reserved words in EventLogging whitelist to unbreak EL sanitization [analytics/refinery] - 10https://gerrit.wikimedia.org/r/451616 (https://phabricator.wikimedia.org/T200095) [12:14:35] (03CR) 10Mforns: [V: 032 C: 032] "Self-merge to unbreak production EL sanitization job" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/451616 (https://phabricator.wikimedia.org/T200095) (owner: 10Mforns) [12:48:26] 10Analytics: Piwik user account for Wikimedia.org.il - https://phabricator.wikimedia.org/T199046 (10Milimetric) @Framawiki: From what I know about our Wikimedia cloud, instances on there, and privacy policies, nothing would prevent you from hosting piwik and sending stats to it from a third party. We don't want... [13:29:32] hola hola [13:40:24] (03PS7) 10Ottomata: Add email error reporting to CamusPartitionChecker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/450861 (https://phabricator.wikimedia.org/T198908) [13:40:34] (03CR) 10Ottomata: "Done." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/450861 (https://phabricator.wikimedia.org/T198908) (owner: 10Ottomata) [13:44:42] (03CR) 10Joal: [C: 031] "LGTM :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/450861 (https://phabricator.wikimedia.org/T198908) (owner: 10Ottomata) [14:06:58] 10Analytics, 10Analytics-EventLogging, 10MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), 10Patch-For-Review, 10Performance-Team (Radar): Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207 (10Krinkle) @ori I'll look into it more later,... [14:22:45] 10Analytics: pyspark2 job killed by YARN for exceeding memory limits - https://phabricator.wikimedia.org/T201519 (10diego) @JAllemandou , I've dona a copy/paste of your code in the Notebook and get same error than before: ``` Py4JJavaError: An error occurred while calling o60.showString. : org.apache.spark.Spar... [14:30:44] 10Analytics: pyspark2 job killed by YARN for exceeding memory limits - https://phabricator.wikimedia.org/T201519 (10JAllemandou) >>! In T201519#4491661, @diego wrote: > @JAllemandou , I've dona a copy/paste of your code in the Notebook and get same error than before: This is expected since there have been no ch... [14:34:13] (03PS5) 10Mforns: Add saltrotate, a script that manages cryptographic salts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/449249 (https://phabricator.wikimedia.org/T199899) [14:34:54] nuria_, yt? can you please review the modifications you suggested ^ ? I already tested them [14:35:05] mforns: on meeting will do after standup [14:35:10] thanks! [14:51:07] o/ milimetric [14:51:15] Thanks for your last comment on T200297 [14:51:16] T200297: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 [14:51:17] hi halfak [14:51:43] We're dying to know what that limit is and I think you have framed the discussion nicely :) [14:52:15] halfak: yeah, I think there's a big issue here, scaling the Revision table, but I think it separate from what you're building [14:52:34] it's very nuanced so in typical Dan fashion I smacked it with a sledgehammer to see if we could simplify it [14:53:09] milimetric, totally. Also, it's not clear to me that the revision table is the biggest issue. I've been hearing that it's actually the page table that is the problem. :S We're very confused about what the actual problem *is*! [14:53:35] Right now, our best understanding is "more is bad" [14:53:44] halfak: they're both growing too fast, and they're causing real problems, and maintenance headaches [14:54:04] and Timo made a great point that new tools like this might hide a little bit the fact that you're actually creating a page behind the scenes [14:54:25] so maybe people will treat it less carefully than the "Create a page" buttons [14:55:05] but, still, they're growing with or without JADE (or RADE!!) and that's why I asked my question [14:55:40] Right. What, exactly, is the problem with page? What sort of maintenance issues are there? How much is too much? We'd like to have a maintenance plan, but we don't have a good sense for what we're dealing with. [14:56:01] if normally we're adding X million records per month, how much can we allow in addition to that and still be ok? If we're ok for 12 months at X million, for example, then we can be ok for 11.9 months at X + y million for a small enough y, and so I don't think that's a concern [14:56:05] Anyway, this is a conversation for another channel. I just wanted to show up here to say "Thanks" because there's no thank button in phab yet :) [14:56:26] ha, oh no prob, yeah, just doing my part as a newbie member of Tech Com [15:01:33] ping joal hola standdupp [15:01:46] joining [15:02:23] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Spark Jupyter Notebook integration - https://phabricator.wikimedia.org/T190443 (10Ottomata) [15:02:24] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Ottomata) [15:03:10] 10Analytics-Kanban, 10Patch-For-Review: Rename column user_name to user_text in user_history for naming coherence - https://phabricator.wikimedia.org/T197926 (10JAllemandou) [15:03:11] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Vet calculation of total article count by summing pages created (with proper filters) over timespam - https://phabricator.wikimedia.org/T199734 (10JAllemandou) [15:03:28] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Ottomata) Could/should we just add another endpoint? /v3/scores-normalized? or even a parameter e.g. /v3/scores?normali... [15:03:43] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add Mediawiki-History data-quality check stage in oozie using statistics - https://phabricator.wikimedia.org/T192481 (10JAllemandou) [15:04:14] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add data-quality check on mediawiki-history-reduced before druid indexation - https://phabricator.wikimedia.org/T192483 (10JAllemandou) [15:07:16] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10mobrovac) >>! In T197000#4489652, @Pchelolo wrote: > After a quick h-o with @Ottomata and @JAllemandou we've understood t... [15:09:47] 10Analytics-Kanban, 10Analytics-Wikistats: Vet calculation of total article count by summing pages created (with proper filters) over timespam - https://phabricator.wikimedia.org/T199734 (10JAllemandou) [15:10:23] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Add data-quality check on mediawiki-history-reduced before druid indexation - https://phabricator.wikimedia.org/T192483 (10JAllemandou) [15:10:41] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add data-quality check on mediawiki-history-reduced before druid indexation - https://phabricator.wikimedia.org/T192483 (10JAllemandou) [15:11:07] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add Mediawiki-History data-quality check stage in oozie using statistics - https://phabricator.wikimedia.org/T192481 (10JAllemandou) [15:19:03] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Halfak) Yes. As you might imagine, we strive for consistency both to keep our engineering simple and to not surprise our... [15:22:42] 10Analytics: Add AQS endpoint providing top editors (by number of edits, by net-bytes-diff and abs-bytes diff) - https://phabricator.wikimedia.org/T201617 (10JAllemandou) p:05Triage>03Normal [15:23:12] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Add AQS endpoint providing top editors (by number of edits, by net-bytes-diff and abs-bytes diff) - https://phabricator.wikimedia.org/T201617 (10JAllemandou) a:03JAllemandou [15:24:47] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Add wikistats metric "top-by-edits" - https://phabricator.wikimedia.org/T189620 (10JAllemandou) a:03JAllemandou [15:25:58] 10Analytics-Kanban: Fix mediawiki-history-druid oozie job - https://phabricator.wikimedia.org/T201620 (10JAllemandou) [15:26:07] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Halfak) Also, I feel like it is important to note that ORES is not a MediaWiki-specific technology. We've been working o... [15:34:28] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Improve scoping of CSS - https://phabricator.wikimedia.org/T190915 (10sahil505) [15:34:31] 10Analytics-Kanban, 10Outreach-Programs-Projects, 10Google-Summer-of-Code (2018): [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189210 (10sahil505) [15:34:51] 10Analytics: Update superset (we have 0.20.6, 0.26.3 is available) - https://phabricator.wikimedia.org/T201430 (10Milimetric) p:05Triage>03High [15:35:17] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Virtual pageview refine should not refine data that does not come from wikimedia domains - https://phabricator.wikimedia.org/T197971 (10Milimetric) p:05Triage>03High [15:35:22] 10Analytics, 10Analytics-EventLogging, 10MW-1.32-release-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)), 10Patch-For-Review, 10Performance-Team (Radar): Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207 (10Milimetric) p:05Triage>03High [15:35:47] 10Analytics, 10Easy: Rename "new pages" endpoint to "net new pages" to better convey that we are reporting a calculation of pages created- pages deleted - https://phabricator.wikimedia.org/T201425 (10Milimetric) p:05Triage>03Normal [15:37:59] 10Analytics, 10Easy: Rename "new pages" endpoint to "net new pages" to better convey that we are reporting a calculation of pages created- pages deleted - https://phabricator.wikimedia.org/T201425 (10Milimetric) 05Open>03declined I prefer to keep the "how" of the metric in the metric definitions and keep t... [15:38:56] 10Analytics: Page creation data no longer updates - https://phabricator.wikimedia.org/T201420 (10Milimetric) p:05Triage>03High [15:39:17] 10Analytics, 10Analytics-Kanban: Page creation data no longer updates - https://phabricator.wikimedia.org/T201420 (10Milimetric) a:03Milimetric [15:45:50] 10Analytics, 10Multimedia: Add mediacounts to pageview API - https://phabricator.wikimedia.org/T88775 (10Milimetric) p:05Lowest>03Normal [15:46:14] 10Analytics: How to get display statistics of the content publised on Commons - https://phabricator.wikimedia.org/T201180 (10Milimetric) Basic usage stats of media is available via our mediacounts dataset (https://dumps.wikimedia.org/other/mediacounts/). We have a task to add this to the pageview API so it's ea... [15:47:50] 10Analytics, 10Analytics-Wikistats: Wikistats 2.0: allow to view stats for all language versions (a.k.a. Project families) - https://phabricator.wikimedia.org/T188550 (10Milimetric) [15:47:53] 10Analytics, 10Analytics-Wikistats: Underreporting WikiSource edits? - https://phabricator.wikimedia.org/T201177 (10Milimetric) 05Open>03Resolved a:03Milimetric Sounds like this is resolved, thanks @Samwalton9 [15:48:38] 10Analytics, 10Analytics-Wikistats: Getting historical country data for WikiSource - https://phabricator.wikimedia.org/T201176 (10Milimetric) 05Open>03Resolved a:03Milimetric [15:51:30] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Add AQS endpoint providing top editors (by number of edits, by net-bytes-diff and abs-bytes diff) - https://phabricator.wikimedia.org/T201617 (10Milimetric) p:05Normal>03High [15:51:45] 10Analytics: pyspark2 job killed by YARN for exceeding memory limits - https://phabricator.wikimedia.org/T201519 (10Milimetric) p:05Normal>03High [15:51:53] 10Analytics, 10Analytics-Kanban: pyspark2 job killed by YARN for exceeding memory limits - https://phabricator.wikimedia.org/T201519 (10Milimetric) p:05High>03Normal a:03Ottomata [15:52:09] 10Analytics: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10Milimetric) p:05Normal>03High [15:52:12] 10Analytics: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10Milimetric) p:05High>03Normal [15:52:15] 10Analytics: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10Milimetric) p:05Normal>03High [15:52:18] 10Analytics: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10Milimetric) p:05High>03Normal [15:56:34] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964 (10sahil505) [15:59:06] (sorry for the spam, I was just moving tasks around on a prioritized board) [15:59:11] * milimetric off to lunch [16:10:44] nice little trick: https://twitter.com/jakevdp/status/1027298136178319360 [16:16:37] 10Quarry, 10Patch-For-Review: Upgrade Quarry to python 3 - https://phabricator.wikimedia.org/T192698 (10zhuyifei1999) @Framawiki Can I play with your `quarry-frama-test-py3-*` instances? (ssh in, install packages, git fetch & checkout, restart services) Kinda lazy to rebuild all these instances. [16:17:10] (03CR) 10Nuria: [V: 032 C: 032] Add saltrotate, a script that manages cryptographic salts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/449249 (https://phabricator.wikimedia.org/T199899) (owner: 10Mforns) [16:17:56] mforns: saltrotate looks (very) good to me, we have tested the loggimg such we can put it in a cron w/o getting spammed right? [16:18:33] mforns: we can merge if so, [16:18:48] nuria_, I remember having tested that, but will check again [16:18:52] ok [16:19:07] mforns: k, looks real simple , nice job [16:19:37] whenever we have to, it will be also simple to keep old salts for a couple weeks [16:19:56] I mean by modifying this script [16:23:24] mforns: ya ya [16:23:38] mforns: understood, for now thins are fine as they are [16:25:34] mforns: FYI that if you add a salt by hand now to teh selected path this code can be started and merged: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/450588/ [16:26:21] mforns: so merging this second patch does not require the saltrotate cron to be productionized [16:26:48] nuria_, but we need to change the cron job that launches the EL san. to add the salt path to it [16:27:22] mforns: ah sorry [16:27:22] no, agree [16:27:26] mforns: got it [16:27:48] in any case, it's not dependant on saltrotate being productionized [16:29:03] nuria_, yes: debug and info go to stdout, warn and error go to stderr [16:29:10] mforns: all right [16:29:12] will merge [16:29:33] mforns: just did [16:29:39] oh ok, thanks! [16:29:55] will deploy then [16:31:38] mforns: I think we could do a first run of sanitization by manually adding a salt just to test [16:32:14] nuria_, I'm currently sanitizing all appInstallID schemas with hash and salt (with a "manually" created salt) [16:32:26] mforns: great [16:32:48] it works just fine (after the yaml reserved words problem) [16:38:12] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Reading-analysis, 10Patch-For-Review: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828 (10Nuria) @chelsyx thanks for working on this. Will review as is but let me outline an approach that... [16:46:18] a-team, starting to deploy refiery-source and refinery [16:46:58] !log starting refiery-source deploy using jenkins [16:47:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:48:27] 10Quarry, 10Patch-For-Review: Upgrade Quarry to python 3 - https://phabricator.wikimedia.org/T192698 (10Framawiki) Hello @zhuyifei1999, these old instances have really been draft :) you'd better create them again. We still have to work on the dependencies and update puppet with this. [[ https://wikitech.wikime... [16:49:44] (03PS1) 10Mforns: Update changelog.md for version v0.0.68 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/451658 [16:50:18] (03CR) 10Mforns: [V: 032 C: 032] "Merging for deployment" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/451658 (owner: 10Mforns) [16:51:11] 10Analytics: Piwik user account for Wikimedia.org.il - https://phabricator.wikimedia.org/T199046 (10Framawiki) >>! In T199046#4491424, @Milimetric wrote: > @Framawiki: From what I know about our Wikimedia cloud, instances on there, and privacy policies, nothing would prevent you from hosting piwik and sending st... [16:51:30] oh mforns ahh! [16:51:32] i wanted to get somethign in [16:51:34] too late eh? [16:51:46] https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/450861/ [16:51:50] oh ottomata, I just merged the changelog changes, I can still add stuff [16:51:58] ok great, you hven't deploye dyet? [16:51:58] cool [16:52:00] i'll rebase and merge [16:52:34] k [16:52:58] chelsyx: yt? [16:53:12] (03PS8) 10Ottomata: Add email error reporting to CamusPartitionChecker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/450861 (https://phabricator.wikimedia.org/T198908) [16:53:28] (03CR) 10Ottomata: [V: 032 C: 032] Add email error reporting to CamusPartitionChecker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/450861 (https://phabricator.wikimedia.org/T198908) (owner: 10Ottomata) [16:53:37] mforns: ok merged. [16:53:47] ottomata, thanks! [16:53:54] will include in the changelog [16:55:28] (03PS1) 10Mforns: Re-update changelog.md for version v0.0.68 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/451660 [16:55:34] (03CR) 10jerkins-bot: [V: 04-1] Re-update changelog.md for version v0.0.68 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/451660 (owner: 10Mforns) [16:58:04] ottomata, oh, you already did add an item to the changelog, right? [16:58:30] nuria_: yes [16:59:33] (03Abandoned) 10Mforns: Re-update changelog.md for version v0.0.68 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/451660 (owner: 10Mforns) [16:59:45] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201063 (10Ottomata) [17:01:03] thx [17:12:22] (03CR) 10Nuria: Add mobile_apps_uniques_by_country_daily and mobile_apps_uniques_by_country_monthly jobs in oozie (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/451566 (https://phabricator.wikimedia.org/T186828) (owner: 10Chelsyx) [17:15:18] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10mobrovac) >>! In T197000#4491764, @Halfak wrote: > From our point of view, you're asking for us to implement and maintain... [17:19:10] chelsyx: sorry, looked at code, let me know if review makes sense [17:20:20] nuria_: sorry I'm in the middle of something. I will reply you on ticket [17:23:31] !log finished refiery-source deploy using jenkins [17:23:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:24:17] !log starting refinery deploy using scap [17:24:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:29:45] 10Analytics: Scan npm dependencies for vulnerabilities - https://phabricator.wikimedia.org/T200717 (10Jdforrester-WMF) [17:35:16] 10Analytics: Scan npm dependencies for vulnerabilities - https://phabricator.wikimedia.org/T200717 (10Legoktm) [17:49:09] !log finished refinery deploy using scap and refinery-deploy-to-hdfs [17:49:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:54:33] 10Quarry, 10Patch-For-Review: Upgrade Quarry to python 3 - https://phabricator.wikimedia.org/T192698 (10zhuyifei1999) >>! In T192698#4492188, @Framawiki wrote: > Hello @zhuyifei1999, these old instances have really been draft :) you'd better create them again. Ok sure. >>! In T192698#4492188, @Framawiki wrot... [17:56:16] 10Quarry, 10Patch-For-Review: Upgrade Quarry to python 3 - https://phabricator.wikimedia.org/T192698 (10zhuyifei1999) Umm. the quota on the # of instances can't let me create more instances... Can I delete them? [18:13:33] 10Analytics, 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: [Investigate] Use PMML for prediction model serialization - https://phabricator.wikimedia.org/T173244 (10awight) [18:18:23] 10Analytics, 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: [Investigate] Use PMML for prediction model serialization - https://phabricator.wikimedia.org/T173244 (10awight) The toolchain isn't very mature. Writing PMML relies on a Java binary, which is acceptable for a compilation pipel... [18:18:34] 10Analytics, 10revscoring, 10Scoring-platform-team (Current), 10artificial-intelligence: [Investigate] Use PMML for prediction model serialization - https://phabricator.wikimedia.org/T173244 (10awight) [18:42:02] nuria_: btw, i was wrong about that hortonworks schema registry [18:42:04] it is not nifi [18:42:06] it has nifi integration [18:42:10] but is not nifi at all [18:42:12] could be interesting [18:42:53] 10Analytics, 10revscoring, 10Scoring-platform-team (Current), 10artificial-intelligence: [Investigate] Use PMML for prediction model serialization - https://phabricator.wikimedia.org/T173244 (10awight) a:03awight [19:42:32] 10Analytics, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000 (10Pchelolo) Let's get back on track. 1. We've discussed the solution of the reformatter in ORES with @Halfak and we decide... [19:45:45] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Reading-analysis, 10Patch-For-Review: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828 (10Tbayer) >>! In T186828#4492134, @Nuria wrote: > @chelsyx thanks for working on this. Will review a... [20:29:32] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry / Metadata Service - https://phabricator.wikimedia.org/T201643 (10Ottomata) p:05Triage>03Normal [20:32:32] (03PS4) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [20:33:45] (03PS5) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [20:43:30] (03PS1) 10Ottomata: Add spark yarn scala and pyspark 'large' kernels [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/451781 (https://phabricator.wikimedia.org/T201519) [20:51:14] (03PS6) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [20:53:29] joal: yt? [20:55:07] (03PS7) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [20:55:58] (03PS1) 10Ottomata: CamusPartitionChecker - only send emails if errors are encountered [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/451784 (https://phabricator.wikimedia.org/T198908) [20:56:31] (03PS2) 10Ottomata: CamusPartitionChecker - only send emails if errors are encountered [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/451784 (https://phabricator.wikimedia.org/T198908) [20:56:39] (03CR) 10Zhuyifei1999: "I'll test it a bit (could you do as well?), and if it works fine we can merge it and get the Python 3 porting going." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [21:06:52] (03PS8) 10Zhuyifei1999: Update dependencies [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/428140 (https://phabricator.wikimedia.org/T192731) (owner: 10Framawiki) [22:10:37] (03PS1) 10QChris: Add .gitreview [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/451796 [22:10:40] (03CR) 10QChris: [V: 032 C: 032] Add .gitreview [analytics/wmde/WiktionaryCognateDashboard] - 10https://gerrit.wikimedia.org/r/451796 (owner: 10QChris) [22:15:21] 10Analytics, 10ORES, 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: [Investigate] Use PMML for prediction model serialization - https://phabricator.wikimedia.org/T173244 (10awight) p:05Normal>03Low Defining fields and mappings should be an interesting exercise. I'm going to deprio... [22:47:51] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Upgrade spark 2.3.0 -> 2.3.1 on analytics cluster - https://phabricator.wikimedia.org/T200732 (10EBernhardson) While i'm not sure what exactly, it seems something might have been missed? Starting an oozie workflow with ``` 10Analytics, 10Datasets-General-or-Unknown, 10Documentation: Missing documentation for pageviews dataset - https://phabricator.wikimedia.org/T201653 (10Tgr) [23:39:19] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Reading-analysis, 10Patch-For-Review: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828 (10chelsyx) @Nuria Thanks for the suggestion and @Tbayer thanks for the insights. > we could have cr...