[00:07:07] 10Analytics, 10Product-Analytics: Event counts from Mysql and Hive don't match. Refine is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Nuria) a:05Nuria→03None [00:15:56] 10Analytics, 10Analytics-Data-Quality, 10Contributors-Analysis, 10Product-Analytics: mediawiki_history missing page events - https://phabricator.wikimedia.org/T205594 (10Nuria) ping @Milimetric @joal @fdans so we have this in our radar for data quality [07:54:53] restarting cassandra on aqs1004 to test the new openjdk version [07:55:00] (only cassandra-a for the moment) [08:52:32] helloooo elukey: you want to try the superset patch at any point this morning? [08:53:17] fdans: sure [08:53:31] hey folllllks [08:53:50] fdans: can you merge the patch so the repo becomes ready to be built? [08:54:20] elukey: yesss [08:54:44] elukey: merged [08:54:48] hellooo mforns [09:21:20] fdans: I am still working on puppet stuff sorry, will try to build the superset repo asap [09:22:08] in theory IIUC Andrew said that we should be ok in changing frozen-requirements.txt in the repo with the link of the github repository [09:22:15] and then run the build script [09:39:47] yea yea elukey no rush, whenever you can, no rush :) [10:02:28] (03PS1) 10Elukey: Revert "Release 0.28.1" [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/480450 [10:03:17] (03CR) 10Elukey: [V: 03+2 C: 03+2] Revert "Release 0.28.1" [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/480450 (owner: 10Elukey) [10:07:28] fdans: what release did we use as baseline for incubator-superset/wikimedia? [10:08:13] 0.26.3 it seems [10:08:31] all right then I don't have the right build string for frozen-requirements.txt [10:08:41] because I am getting 0.28.1 deps [10:08:54] I think that it is not getting the 'wikimedia' branch [10:12:28] I am basically trying [10:12:28] -superset==0.26.3 [10:12:28] +git+https://github.com/wikimedia/incubator-superset.git@wikimedia [10:13:13] Cloning https://github.com/wikimedia/incubator-superset.git (to revision wikimedia) to /tmp/pip-req-build-z8lfr87b [10:13:16] Branch wikimedia set up to track remote branch wikimedia from origin. [10:13:19] Switched to a new branch 'wikimedia' [10:13:19] that seems ok [10:13:43] is that output from the console? [10:16:50] yep, from build [10:17:31] I am going to send a cr with the current state of the artifacts [10:17:41] one thing that I am not sure of is this [10:18:00] https://github.com/wikimedia/incubator-superset/blob/wikimedia/pypi_push.sh [10:18:10] that was basically one of my steps [10:19:25] !log restarted Turnilo to clear deleted datasource [10:19:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:19:27] I suspect that if we add the github repo to frozen-requirements.txt and build then we'll skip the npm stuff [10:21:22] (03PS1) 10Elukey: [WIP] Release 0.26.3 (patched) [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/480455 [10:22:01] this is the result --^ [10:23:17] I would have expected, in theory, no changes in artifacts [10:23:23] (except superset) [10:25:09] let's quickly check https://github.com/apache/incubator-superset/blob/0.26/requirements.txt [10:27:11] so assuming that this is "expected" by pip install since it grabs the latest deps available (the ones without a strict version specified, like the requirements.txt's deps) [10:27:23] I am a little worried about https://gerrit.wikimedia.org/r/#/c/analytics/superset/deploy/+/480455/1/artifacts/stretch/MarkupSafe-1.1.0-cp35-cp35m-manylinux1_x86_64.whl,unified [10:27:42] fdans: do you remember the broken charts issue? IIRC it was due to --^ [10:27:49] but let's deploy in labs and see [10:28:19] hmmm, yeah let's try [10:29:49] heya, making a pause, will be back later! [10:52:16] fdans: so I am now getting this while trying to upgrade the db [10:52:17] https://github.com/apache/incubator-superset/issues/5958 [10:52:18] ahahahah [10:52:59] agghhhh lol [10:54:19] wait something weird is happening [10:54:51] ah no it makes complete sense [10:54:53] D artifacts/stretch/Markdown-2.6.11-py2.py3-none-any.whl -76.6 KiB (-100%) [10:54:56] A artifacts/stretch/Markdown-3.0.1-py2.py3-none-any.whl +87.3 KiB [10:55:02] so yeah artifacts are broken [10:55:03] iff [10:55:07] *uff [10:55:34] going back to build [11:00:23] (03Abandoned) 10Elukey: [WIP] Release 0.26.3 (patched) [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/480455 (owner: 10Elukey) [11:02:13] ok so I am probably stupid and missing something [11:02:17] https://github.com/wikimedia/incubator-superset/blob/wikimedia/requirements.txt is clear [11:02:36] markdown 2.6.11, that is the one that we want [11:03:01] Collecting markdown (from superset==0.26.3->-r ./frozen-requirements.txt (line 1)) Downloading https://files.pythonhosted.org/packages/7a/6b/5600647404ba15545ec37d2f7f58844d690baf2f81f3a60b862e48f29287/Markdown-3.0.1-py2.py3-none-any.whl (89kB) [11:03:09] whattttt [11:05:28] so it seems that it grabs deps from master [11:11:03] elukey: would it help to brainbounce? [11:16:19] sure [11:21:50] maybe it is better to just wait for andrew [11:24:02] https://github.com/pypa/pip/issues/3610#issuecomment-239940024 [11:30:12] yes let's wait Andrew [11:30:15] ok fdans ? [11:30:20] he might have already a solution [11:30:26] I am a bit lost [11:30:33] elukey: sounds good [12:04:53] * elukey lunch! [12:06:28] (03PS1) 10Fdans: Remove sessionID from whitelist for MobileWebSectionUsage schema [analytics/refinery] - 10https://gerrit.wikimedia.org/r/480470 (https://phabricator.wikimedia.org/T209049) [12:09:33] (03PS1) 10Fdans: Remove pageTitle from whitelist for Print schema [analytics/refinery] - 10https://gerrit.wikimedia.org/r/480471 (https://phabricator.wikimedia.org/T209050) [12:12:10] (03PS1) 10Fdans: Remove session token from whitelist for ReadingDepth schema [analytics/refinery] - 10https://gerrit.wikimedia.org/r/480472 (https://phabricator.wikimedia.org/T209051) [14:25:56] 10Analytics: Clean up home dirs for users jamesur and nithum - https://phabricator.wikimedia.org/T212127 (10fdans) Stuff to delete in both users jamesur: - home dir in stat1005 - home dir in hdfs - database `jamesur` in hive (1 table) nithum: - home dir in stat1004 - home dir in stat1005 - home dir in stat1... [15:15:17] joal: how goes in presto land? (saw your email) [15:18:44] o/ [15:21:19] ottomata: hiiiiiiiii [15:21:37] whenever you have a minute I'd have some ideas to share in bc [15:21:40] for next Q [15:34:30] !log restarted Turnilo to clear deleted datasource [15:34:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:35:34] elukey: hiiiii i am at a cafe! [15:35:44] after retro ok? [15:38:53] sure :) [15:39:30] * elukey afk for 10 mins [15:59:44] (03PS2) 10Michael Große: Add build for deployment [analytics/wmde/toolkit-analyzer-build] - 10https://gerrit.wikimedia.org/r/480036 (https://phabricator.wikimedia.org/T209399) [16:01:19] fdans, mforns: standduppp [16:02:15] joinin [16:17:51] PROBLEM - cache_text: Varnishkafka Webrequest Delivery Errors per second on icinga1001 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [5.0] https://grafana.wikimedia.org/dashboard/db/varnishkafka?panelId=20fullscreenorgId=1var-instance=webrequestvar-host=All [16:20:07] PROBLEM - cache_upload: Varnishkafka Webrequest Delivery Errors per second on icinga1001 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [5.0] https://grafana.wikimedia.org/dashboard/db/varnishkafka?panelId=20fullscreenorgId=1var-instance=webrequestvar-host=All [16:21:07] ah yes this is Arzhel doing maintenance in codfw probably [16:23:32] seems recovered [16:25:07] RECOVERY - cache_text: Varnishkafka Webrequest Delivery Errors per second on icinga1001 is OK: OK: Less than 1.00% above the threshold [1.0] https://grafana.wikimedia.org/dashboard/db/varnishkafka?panelId=20fullscreenorgId=1var-instance=webrequestvar-host=All [16:26:09] RECOVERY - cache_upload: Varnishkafka Webrequest Delivery Errors per second on icinga1001 is OK: OK: Less than 1.00% above the threshold [1.0] https://grafana.wikimedia.org/dashboard/db/varnishkafka?panelId=20fullscreenorgId=1var-instance=webrequestvar-host=All [16:26:34] * elukey likes the new super sensitive alarms [16:31:58] 10Analytics-Kanban, 10User-Elukey: Q1 2018/19 Analytics procurement - https://phabricator.wikimedia.org/T198694 (10RobH) [16:32:01] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad | (14 + 6) hadoop hardware refresh and expansion - https://phabricator.wikimedia.org/T199673 (10RobH) 05Open→03Resolved a:03RobH This has been filled via #procurement task T204177, resolving. [17:09:15] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10Ottomata) Hey just started following this ticket. BTW, I really like the idea of including the sample rate in the event. The Better Use of Data working group has be... [17:16:48] 10Analytics, 10Analytics-EventLogging: EventLogging client side serialization saves integer decimals as decimal-less numbers - https://phabricator.wikimedia.org/T211983 (10Ottomata) Another thought (prompted by Dan). If we were creating the Hive schemas directly from the JSON Schemas (as we hope to do with Ka... [17:37:23] !log restarted Turnilo to clear deleted datasource [17:37:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:52:25] running some errands, back in a bit! [17:54:16] 10Analytics, 10Analytics-Kanban, 10Fundraising-Backlog, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10DStrine) [18:07:03] * elukey off! [18:21:47] (03CR) 10Mforns: [C: 04-2] Allow for custom transforms in DataFrameToDruid (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099) (owner: 10Mforns) [18:31:25] mforns: just read your comment response [18:31:29] trying to understand [18:31:31] ottomata, [18:31:43] wouldn't that be the case anyway? if the caller of DataFrameToDruid provides flatteners? [18:31:47] if you want we can bc, I was writing on top of that one :] [18:31:52] ok! [18:31:57] ok omw [18:31:57] ya let's bc [18:35:57] (03PS11) 10Mforns: Allow for custom transforms in DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/477295 (https://phabricator.wikimedia.org/T210099) [18:42:09] 10Analytics: Staging environment for upgrades of superset - https://phabricator.wikimedia.org/T212243 (10Nuria) [18:42:35] 10Analytics: Staging environment for upgrades of superset - https://phabricator.wikimedia.org/T212243 (10Nuria) [18:42:37] 10Analytics-Kanban, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10Nuria) [18:51:01] 10Analytics, 10Analytics-Kanban: Port our "fixes" for python 3.5 to our superset fork https://github.com/wikimedia/incubator-superset - https://phabricator.wikimedia.org/T211932 (10Nuria) I am closing this task as we would not need to port fixes to 3.5 since we will be able to run in 3.6. It is likely we need... [18:51:14] 10Analytics, 10Analytics-Kanban: Port our "fixes" for python 3.5 to our superset fork https://github.com/wikimedia/incubator-superset - https://phabricator.wikimedia.org/T211932 (10Nuria) 05Open→03Declined [18:51:16] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10Nuria) [18:53:40] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Nuria) I can see this is fixed for 11/30 but I think refine of data is still running. [18:56:54] ottomata: i think the re-refinment of data is going well right? [19:04:26] nuria: it is still running! checking that its not stuck... [19:06:08] nuria: i think it is not going well, i still see nulls....... [19:06:22] and the spark ui is not resoponding via yarn [19:06:27] so i can't tell what it is doing... [19:06:30] investigating... [19:06:35] ottomata: on meeting, can check in a bit , [19:07:13] ottomata: this returns good data when before it didn't [19:07:16] https://www.irccloud.com/pastebin/eZnx6Lqf/ [19:07:45] yeah, but later data doesn't...i wonder if my original backfill (last week when I deleted _REFINE flags fixed that) [19:10:49] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 (10Ottomata) Yeah, the backfill job is still running but seems stuck. Investigating... [19:14:20] nuria: hah i can see why my re-refine from last week failed...typo in script i wrote to delete _REFINE flags [19:14:25] it didn't remove the december ones [19:14:32] which is why the 11/30 one is fine [19:17:24] yeah it just looks stuck... [19:17:26] it was working tho [19:17:29] some of the data finished [19:17:31] i'm going to kill this job [19:17:34] and run some smaller ones [19:17:40] instead of one huge backfill job [19:36:49] !log re-running refine_eventlogging_backfill again for days in december - T211833 [19:36:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:36:56] T211833: [BUG] User agent parsing error for MobileWikiAppSearch table - https://phabricator.wikimedia.org/T211833 [20:15:23] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: [BUG] userAgent missing from all EventLogging analytics Hive tables between 2018-11-30 and 2018-11-14 - https://phabricator.wikimedia.org/T211833 (10Ottomata) [20:15:31] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: [BUG] userAgent missing from all EventLogging analytics Hive tables between 2018-11-29 and 2018-11-14 - https://phabricator.wikimedia.org/T211833 (10Ottomata) [20:18:34] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: [BUG] userAgent missing from all EventLogging analytics Hive tables between 2018-11-29 and 2018-11-14 - https://phabricator.wikimedia.org/T211833 (10Ottomata) The days in 2018-11 are done. Here's how I launched a re-refine for days... [20:24:51] Gone for tonight team - MWH job succeeeded, but there still is an issue :( Data is copying on ca-hadoop for presto [20:26:22] oook ! [20:26:27] laters joal! [20:34:50] !log bounced eventlogging-processor to pick up change to send invalid rawEvents as json string [20:35:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:47:38] 10Analytics, 10Fundraising-Backlog: Identify source of discrepancy between HUE query in Count of event.impression and druid queries via turnilo/superset - https://phabricator.wikimedia.org/T204396 (10DStrine) Quick update here. We are still using the older pipeline for this season so this is not affecting us a... [20:47:46] 10Analytics, 10Fundraising-Backlog: Identify source of discrepancy between HUE query in Count of event.impression and druid queries via turnilo/superset - https://phabricator.wikimedia.org/T204396 (10DStrine) p:05High→03Normal [20:47:56] 10Analytics, 10Analytics-Kanban, 10Fundraising-Backlog, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10DStrine) p:05High→03Normal [20:48:04] (03CR) 10Nuria: [C: 03+1] Add ServerTiming to EventLogging whitelist (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476841 (https://phabricator.wikimedia.org/T207862) (owner: 10Gilles) [20:49:53] 10Analytics, 10Fundraising-Backlog: Identify source of discrepancy between HUE query in Count of event.impression and druid queries via turnilo/superset - https://phabricator.wikimedia.org/T204396 (10Nuria) @DStrine can you clarify what is the old pipeline? [20:52:41] 10Analytics, 10Analytics-Kanban, 10Fundraising-Backlog, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10DStrine) This has not been mission critical for us. Fr-tech will get back to this in the new year. [21:00:01] huh! milimetric have you seen https://eventmetrics.wmflabs.org ? [21:01:09] ottomata: yeah, that’s the project we hope replaces wikimetrics, we’re going to help build better apis for them to use [21:01:31] Leon’s working on it [21:06:27] cool! [21:06:31] yeah i just saw it [21:11:55] 10Analytics, 10Fundraising-Backlog: Identify source of discrepancy between HUE query in Count of event.impression and druid queries via turnilo/superset - https://phabricator.wikimedia.org/T204396 (10DStrine) Kafkatee @AndyRussG assures us this would not have impacted our campaign stats. [21:31:25] 10Analytics: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one. - https://phabricator.wikimedia.org/T212256 (10Nuria) [21:32:06] 10Analytics: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one. - https://phabricator.wikimedia.org/T212256 (10Nuria) [21:32:08] 10Analytics: Kerberos stand alone service running in production - https://phabricator.wikimedia.org/T211836 (10Nuria) [21:32:48] 10Analytics: Set up a Kerberos KDC service in production with minimal puppet automation - https://phabricator.wikimedia.org/T212257 (10Nuria) p:05Triage→03High [21:33:22] 10Analytics: Create test Kerberos identities/accounts for some selected users from Analytics - https://phabricator.wikimedia.org/T212258 (10Nuria) p:05Triage→03High [21:33:51] 10Analytics: Run critical Analytics Hadoop jobs and make sure that they work with the new auth settings. - https://phabricator.wikimedia.org/T212259 (10Nuria) p:05Triage→03High [21:37:19] 10Analytics: Kerberos service running in production - https://phabricator.wikimedia.org/T211836 (10Nuria) [21:40:07] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: [BUG] userAgent missing from all EventLogging analytics Hive tables between 2018-11-29 and 2018-11-14 - https://phabricator.wikimedia.org/T211833 (10Ottomata) Alright! Looking waayyy better this time. Backfilling should be done. @m... [21:41:19] 10Analytics: Kerberos service running in production - https://phabricator.wikimedia.org/T211836 (10Nuria) [21:43:34] 10Analytics, 10User-Elukey: Prototype in labs new security measures for cluster - https://phabricator.wikimedia.org/T198227 (10Nuria) We have successful doing the prototyping in labs in Q2. We are closing this task and moving to deploying kerberos in prod, initial steps in this regard will start q3. See par... [21:43:43] 10Analytics, 10User-Elukey: Prototype in labs new security measures for cluster - https://phabricator.wikimedia.org/T198227 (10Nuria) 05Open→03Resolved [21:48:21] ottomata: we have a goal of sunseting wikimetrics next quarter , it will redirect to https://eventmetrics.wmflabs.org [21:55:17] ahh very cool [22:16:54] joal: did we moved the presto document to wikitech? [22:18:39] yes nuria - https://wikitech.wikimedia.org/wiki/Analytics/Projects/Data_Lake/SQL_Engine_on_Cloud and https://wikitech.wikimedia.org/wiki/Analytics/Projects/Data_Lake/SQL_Engine_on_Cloud/Appendix [22:19:08] joal: i totally found link on e-mail, let me see if i can find a better title so it is easier to search for it [22:27:15] joal: I have renamed it: https://wikitech.wikimedia.org/wiki/Analytics/Projects/Public_Data_Lake [22:29:25] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) I'd love some advice on how to proceed with this. The new stream's structure will be slightly different than mediawiki.revision-score, the biggest change is that we want... [22:42:09] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) Refinery's json_refine_job seems to fulfill the function of Connect, e.g. https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/387838/7/modules/role/manifests/analytics_... [22:49:52] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Nuria) @awight kafka topics (sometimes called streams) are set by schema and each schema is propagated to a different table in hive. Hopefully this makes sense. This means that ev... [22:51:58] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Ottomata) @awight, I think it would be better to transform the data in event.mediawiki_revision_score to your format in Hadoop. You can do this via Hive or Spark, possibly even r... [22:54:50] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Ottomata) @nuria, we talked about making a new stream for each model a lot in {T197000} but ultimately decided against it. @awight should split the existing data in hive into a n... [23:01:02] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) >>! In T209732#4832862, @Ottomata wrote: > @awight should split the existing data in hive into a new Hive tables by model. Thanks for the suggestions! Would this be a Re... [23:10:18] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Ottomata) I think it depends on what you want to do. Hm, actually, this is not a 'Refine' job. (We use the term 'refine' to mean a 1 to 1 dataset job. Take one dataset in, enhan... [23:19:20] 10Analytics, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Add EditAttemptStep properties to the schema whitelist - https://phabricator.wikimedia.org/T208332 (10Neil_P._Quinn_WMF) 05Resolved→03Open @Nuria when should we see data start flowing into `event_sanitized.editattemptstep`? The table... [23:20:46] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) >>! In T209732#4832890, @Ottomata wrote: > The tricky part with ORES models is that they don't share a common schema. Good point, I guess the key-value array for probabil... [23:26:08] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Ottomata) Ah ok, cool! Good to know @JAllemandou is involved there. I think he can help you figure out how this job/query would look. That partitioning scheme makes sense. If... [23:30:41] 10Analytics, 10ORES, 10Scoring-platform-team: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Nuria) I see, rather that a new stream of data this is a pivot of data we already have so spark job makes sense. [23:32:49] 10Analytics, 10Dumps-Generation, 10ORES, 10Scoring-platform-team, and 3 others: [Epic] Make ORES scores available in Hadoop and as a dump - https://phabricator.wikimedia.org/T209611 (10awight) I'm reducing the scope of this task to just one pilot integration, for wikidata. [23:33:17] 10Analytics, 10Dumps-Generation, 10ORES, 10Scoring-platform-team, and 3 others: [Epic] Make ORES scores for wikidata available as a dump - https://phabricator.wikimedia.org/T209611 (10awight) [23:37:14] 10Analytics, 10Analytics-Kanban, 10Fundraising-Backlog, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10Nuria) I see, @DStrine and @AndyRussG next time around let's please make clear on ticket this is not critical/important, we assumed it was... [23:45:47] 10Analytics, 10Analytics-Kanban, 10Fundraising-Backlog, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10Nuria) @JAllemandou can we keep the code to initiate the real time ingestion in the refinery repo? I think we just need a bit of documentatio... [23:57:48] 10Analytics, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Add EditAttemptStep properties to the schema whitelist - https://phabricator.wikimedia.org/T208332 (10Nuria) >Whitelist changes are applied since the date they get merged, older data is not revised so whatever was sanitized before the mer...