[04:11:29] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue) - https://phabricator.wikimedia.org/T252703 (10Nuria) links: https://superset.wikimedia.org and https://turnilo.wikimedia.org [08:40:30] (03CR) 10Joal: "Another bunch of comments, mostly naming and comments. Thanks fdans" (0313 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) (owner: 10Fdans) [08:46:29] (03CR) 10Joal: "One detail on naming, feels ready otherwise :)" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) (owner: 10Milimetric) [08:52:48] win 28 [08:52:50] uff [08:52:52] morning :) [08:53:28] o/ elukey [08:56:14] (03PS42) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [08:57:24] (03CR) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ (0310 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) (owner: 10Fdans) [08:57:58] joal: bonjour :) I realized that I need to build a new version of druid for the partition change, but I currently have some issues.. when I solve them, I am planning to just put an-druid100[1,2] in service [08:58:02] is it ok? [08:58:12] then we'll see what works and what not [08:58:17] seems relatively low risk [08:58:38] we'll do the public cluster once we'll be confident [08:58:56] works for me elukey - Need to leave in minutes but will be back around 12:30 [09:00:06] ack :) [09:46:07] joal: do you have any idea of what the .mw project family (with language) could be? there's a lot of pagecounts for them [09:47:04] like there's only one row per wiki [09:47:12] and each is a language code: [09:47:22] https://www.irccloud.com/pastebin/HcLFaocs/ [09:48:54] ha, it's mobile wikipedia? [09:49:05] but like, only per project views? [09:49:25] joal: https://phabricator.wikimedia.org/diffusion/ANWC/browse/master/filter.c$98 [09:49:43] (sorry for the spam, though you might be interested but feel free to tell me to stop) [09:56:01] 10Analytics, 10Analytics-Kanban, 10Dumps-Generation: Document missing project types in pagecount dumps - https://phabricator.wikimedia.org/T249984 (10fdans) Unearthing more things, from Webstatscollector's docs: > Requests to mobile sites get aggregated across projects. So “en.mw" does not refer to “PageVi... [10:04:08] 10Analytics: Check home/HDFS leftovers of thargrove - https://phabricator.wikimedia.org/T253000 (10MoritzMuehlenhoff) [10:07:18] it doesn't matter because we're not tracking mobile views before 2016 but oh boy [10:08:26] the pagecounts-ez dumps were perfectly documented in the same way that the pyramids of egypt were perfectly documented with hieroglyphs [10:08:55] the C code of webstatscollector is the rosetta stone in this analogy [10:09:30] ahhahahaha [10:09:46] and a good day to you elukey [10:09:52] :D [10:11:44] good morning :D [10:11:52] I am adding two new druid hosts to the analytics cluster [10:11:55] as FYI [10:12:07] hell yea [10:13:28] (03PS4) 10Awight: Embed Jupyter kernel template [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596421 [10:19:35] (03CR) 10Awight: [V: 03+1 C: 03+2] "Verified that this works with the latest changes. Thanks!" [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225 (owner: 10Awight) [10:19:54] (03PS8) 10Awight: Clean up table persistence [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225 [10:20:03] (03CR) 10Awight: [V: 03+2 C: 03+2] Clean up table persistence [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225 (owner: 10Awight) [10:22:06] (03PS4) 10Awight: Move data to semi-permanent path [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596436 [10:24:14] (03PS4) 10Awight: Remove known buggy conflicts [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596439 [10:24:42] (03PS5) 10Awight: Count displayed rows [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) [10:28:29] I'm back :) [10:33:07] !log add an-druid100[1,2] to the Druid Analytics cluster [10:33:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:38:30] the coordinator seems happy with 5 nodes :) [10:38:53] there is some tuning needed, the new nodes are beefier [10:42:18] also the overlord looks fine [10:43:59] so follow ups: [10:44:19] 1) make druid config somehow dependend on core/memory available on hosts [10:44:52] 2) think about the fact that moving druid nodes from stretch to buster will mean also moving zookeeper [10:45:19] that we run with openjdk-11 on an-conf [10:45:24] so either we move druid to an-conf [10:46:07] or we keep jdk-8/11 on the druid nodes, and see if druid/zookeper can be configured with a specific jdk rather than picking from alternatives [11:06:15] I have also realized that we don't use g1 for overlord/middlemanager/historical, but CMS [11:06:18] mmmmm [11:48:57] * elukey lunch! [12:00:10] (03CR) 10Awight: "I've pushed to a sandbox branch so the notebooks can be reviewed graphically:" [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596480 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [12:01:27] (03CR) 10Awight: "I've pushed to a sandbox branch in case that helps with review. Unfortunately, the graphics won't load there." [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [12:39:54] 10Analytics, 10Operations, 10serviceops, 10vm-requests: Create a VM for matomo1002 (eqiad) - https://phabricator.wikimedia.org/T252742 (10Marostegui) p:05Triage→03Medium [13:07:01] joal: I am refreshing a bit my stale cassandra knowlege, so I checked a bit our schemas. The partition ("_domain", project, article, granularity) of pageviews_per_article_flat is useful to better partition key across nodes and also to get good query perf over range queries for pageview values over a given time window? [13:07:19] elukey: in meeting - will get back to you after [13:07:24] ah yes yes [13:07:26] anytime :) [13:26:49] 10Analytics: Add new Druid nodes to analytics and public clusters - https://phabricator.wikimedia.org/T252771 (10elukey) The new hosts have been added to the Analytics cluster, some notes: * after running puppet on an-druid* both openjdk 11 and 8 were deployed, with 11 selected by alternatives and Druid was not... [13:30:20] !log roll restart hadoop daemons on the test cluster for openjdk upgrades [13:30:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:30:29] the main cluster will follow later on [13:53:50] 10Analytics: Check home/HDFS leftovers of thargrove - https://phabricator.wikimedia.org/T253000 (10elukey) ` ====== stat1004 ====== total 0 ls: cannot access '/var/userarchive/thargrove.tar.bz2': No such file or directory ====== stat1005 ====== total 0 ls: cannot access '/var/userarchive/thargrove.tar.bz2': No... [13:55:41] 10Analytics: Check home/HDFS leftovers of thargrove - https://phabricator.wikimedia.org/T253000 (10elukey) 05Open→03Resolved a:03elukey Cleaned up :) [14:02:46] !log roll restart of hadoop daemons on the prod cluster for openjdk upgrades [14:02:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:05:12] moritzm: hiya, yt? I need to copy confluent-kafka-2.11 from stretch-wikimedia into buster-wikimedia [14:05:26] reprepro -C thirdparty/confluent ls confluent-kafka-2.11 [14:05:26] confluent-kafka-2.11 | 1.1.0-1 | stretch-wikimedia | amd64, i386 [14:05:28] but [14:05:36] reprepro -C thirdparty/confluent copy buster-wikimedia stretch-wikimedia confluent-kafka-2.11 [14:05:36] Will not copy as not found: confluent-kafka-2.11. [14:05:41] but sure what i'm doing wrong there [14:06:40] the copy command goes by source packages, while the Kafka vendor deb only ships the debs, you can use [14:06:46] ah [14:06:52] includedeb? [14:07:18] yeah, simply use includeddeb with the existing package [14:07:19] oh i see the alternate procedure now described at https://wikitech.wikimedia.org/wiki/Reprepro#Copying_between_distributions [14:07:23] ok great [14:07:25] thank ypou [14:08:33] oh [14:08:42] moritzm: i think thirdparty/confluent might not be in buster-wikimedia [14:08:50] do I need to create it first? [14:10:28] (no part of the conversation but I think so, it is not listed in puppet) [14:10:36] (hello :) [14:10:48] (hello!) ya see it making patch [14:12:39] oh and maybe adding it to updates will get it automatically pulled into buster-wikimedia [14:14:20] yeah, if we sync it from an external repo, then let's simply add the definition and sync from there instead of adding it with includedeb [14:14:47] https://gerrit.wikimedia.org/r/c/operations/puppet/+/597076 [14:14:48] moritzm: ^ [14:16:24] hi elukey - ready I am if you want [14:16:56] joal: hello! no problem, it was just a random cassandra question, nothing urgent [14:17:05] np elukey - I can answer :) [14:17:12] elukey: do you wish to batcave for a minute? [14:17:21] joal: sure, gimme a min [14:17:43] 10Analytics, 10Analytics-Kanban, 10Dumps-Generation: Document missing project types in pagecount dumps - https://phabricator.wikimedia.org/T249984 (10Nuria) >So requests to each of http://en.m.wikivoyage.org/wiki/Main_Page, and http://en.m.wikipedia.org/wiki/Main_Page will get counted towards en.mw. Seems un... [14:17:57] (03PS3) 10Elukey: Release upstream version 0.36.0 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/591440 (https://phabricator.wikimedia.org/T249495) [14:21:45] 10Analytics, 10Analytics-Kanban, 10Dumps-Generation: Document missing project types in pagecount dumps - https://phabricator.wikimedia.org/T249984 (10fdans) Oh yeah btw, something I discovered last week: webhosts are not being passed through a toLowerCase function in the current Pagecounts-EZ form, so view c... [14:30:40] thanks moritzm updated [14:30:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Migrate pagecounts-ez generation to hadoop - https://phabricator.wikimedia.org/T192474 (10Nuria) Is this change from @Milimetric now abandoned? https://gerrit.wikimedia.org/r/562079 [14:34:20] left two more comments [14:36:23] ty, patched again mo [14:36:25] moritzm: [14:36:50] ottomata: o/ do we have a retention period for events (like mediawiki.revision-create) stored in hdfs that comes from changeprop? [14:37:05] wait did I lose thirdparty/confluent from buster? onse sec [14:37:54] dcausse: in hdfs? events stay forever, no PII there [14:38:08] dcausse: those are public so retained forever [14:38:22] dcausse: (edit info is public, pageview info is not) [14:38:32] thanks! [14:48:53] moritzm: do I need to make updates run? [14:48:55] or wait? [14:49:12] getting 'buster-wikimedia InRelease' doesn't have the component 'thirdparty/confluent'' on kafka-jumbo1007 [14:49:17] during apt-get update [14:51:03] you need to run puppet on apt1001 to add the component in the repo [14:51:22] for the component itself to be added on kafka-jumbo1007 it depends on the existing puppet setup [14:51:41] if you're using apt::package_from_component is should work transparently [14:52:05] but if it's the old, manual apt repo setup code, you might need a Puppet change to also enable buster [14:52:44] ok we are using apt::repository and pakage [14:52:49] wiwll switch to package_from_component [14:55:09] joal: batcave for a quick minute to wrap up that change? [14:55:18] sure milimetric [15:00:04] 10Analytics, 10Cassandra, 10User-Elukey: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10elukey) @Eevans please be patient, me and Joseph had a long chat about upgrading in place and there are some doubts that we have. Overall, what we'd like to do is something like: 1) sto... [15:08:53] (03PS6) 10Milimetric: Use new page move incremental updates [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) [15:09:24] (03CR) 10Milimetric: "k, this should be ready, thanks for the many reviews!" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) (owner: 10Milimetric) [15:13:00] elukey: were you or ottomata able to update the archiva password? I wasn't sure last week [15:13:17] no i didn't [15:13:31] thought you might want you since i can't seem to use pwstore [15:13:33] i don't know how people use it [15:13:40] folks keys are always expired [15:14:09] oh sorry read that as luca asking me [15:23:12] ottomata: I can try to figure out pwstore [15:34:01] ottomata: you can always simply use "gpg --decrypt archiva" as well [15:34:10] pwstore is just a fancy wrapper [15:37:52] (03CR) 10Milimetric: "documented the datasets initial-instance gotcha: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Oozie#Datasets" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) (owner: 10Milimetric) [16:01:38] 10Analytics, 10Cassandra, 10User-Elukey: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10Eevans) >>! In T249756#6145067, @elukey wrote: > @Eevans please be patient, me and Joseph had a long chat about upgrading in place and there are some doubts that we have. Overall, what w... [16:04:48] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create job that backfills Pagecounts-EZ (2011 - 2016) data via hadoop correcting issues - https://phabricator.wikimedia.org/T252857 (10fdans) p:05Triage→03High [16:06:09] 10Analytics, 10Analytics-Kanban: Add new Druid nodes to analytics and public clusters - https://phabricator.wikimedia.org/T252771 (10fdans) [16:06:36] 10Analytics, 10Discovery-Analysis, 10Patch-For-Review, 10Product-Analytics (Kanban): Decomission maps metrics module from wikimedia/discovery/golden data retrieval - https://phabricator.wikimedia.org/T252365 (10Mayakp.wiki) p:05Medium→03High [16:07:27] 10Analytics, 10Discovery-Analysis, 10Patch-For-Review, 10Product-Analytics (Kanban): Decomission maps metrics module from wikimedia/discovery/golden data retrieval - https://phabricator.wikimedia.org/T252365 (10Mayakp.wiki) p:05High→03Medium [16:08:12] 10Analytics: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete - https://phabricator.wikimedia.org/T252585 (10fdans) p:05Triage→03Medium [16:08:15] 10Analytics, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T252915 (10Milimetric) Looks like some files actually haven't run the April job properly, and there might be a bug with rendering the rest of the data. I'll t... [16:09:13] 10Analytics, 10Analytics-Kanban: Update sqoop before labs views change - https://phabricator.wikimedia.org/T252565 (10fdans) p:05Triage→03High a:03JAllemandou [16:09:53] 10Analytics: [Spike] Explore goblin as an alternative to camus - https://phabricator.wikimedia.org/T252560 (10fdans) p:05Triage→03High a:03Ottomata [16:13:13] 10Analytics, 10Product-Analytics: Update mediawiki_user_blocks_change to log partial block parameters - https://phabricator.wikimedia.org/T252455 (10fdans) p:05Triage→03High [16:15:08] 10Analytics, 10Core Platform Team, 10Product-Analytics: Update mediawiki_user_blocks_change to log partial block parameters - https://phabricator.wikimedia.org/T252455 (10fdans) [16:15:24] 10Analytics, 10Product-Analytics (Kanban), 10User-Elukey: Learn how to make dashboard on top of data on hadoop/hive via presto - https://phabricator.wikimedia.org/T247329 (10cchen) [16:16:01] 10Analytics, 10Core Platform Team, 10Product-Analytics: Update mediawiki_user_blocks_change to log partial block parameters - https://phabricator.wikimedia.org/T252455 (10fdans) cc @WDoranWMF and @Pchelolo events sent by MediaWiki need to be augmented with the new parameters. [16:17:50] 10Analytics: Add Authentication/Encryption to Kafka Jumbo's clients - https://phabricator.wikimedia.org/T250146 (10elukey) 05Open→03Stalled Status update: we added encryption to various Kafka Jumbo clients (netflow, eventgate-analytics, kafkatee, mirror-maker, etc..) but we are still not able to move forward... [16:20:13] 10Analytics: Add a "latest" partition to Hive tables - https://phabricator.wikimedia.org/T252148 (10fdans) p:05Triage→03Medium [16:22:02] 10Analytics, 10User-Elukey: Upgrade Druid to its latest upstream version (currently 0.18.1) - https://phabricator.wikimedia.org/T244482 (10elukey) [16:22:36] 10Analytics: Investigate tools.wmflabs.org to toolforge.org migration - https://phabricator.wikimedia.org/T250116 (10fdans) a:03Milimetric [16:25:13] 10Analytics, 10EventStreams: EventStreams socket stays connected without any traffic incoming - https://phabricator.wikimedia.org/T250912 (10fdans) cc @Ottomata [16:30:22] 10Analytics, 10Product-Analytics (Kanban): Create Druid tables for Druid datasources in Superset - https://phabricator.wikimedia.org/T251857 (10fdans) @cchen just to clarify, is this `eventCount` column that count measure that druid adds to the datasource, or is it a column that should have been ingested as pa... [16:32:44] 10Analytics: Add Authentication/Encryption to Kafka Jumbo's clients - https://phabricator.wikimedia.org/T250146 (10elukey) [16:32:46] 10Analytics: [Spike] Explore goblin as an alternative to camus - https://phabricator.wikimedia.org/T252560 (10elukey) [16:42:36] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) Hio, I see some upgrades happened, should they have fixed this? [16:46:52] 10Analytics, 10Discovery, 10Discovery-Analysis, 10Product-Analytics: Add referer to WebrequestData - https://phabricator.wikimedia.org/T172009 (10mpopov) 05Open→03Declined [17:03:56] * elukey off! [17:23:30] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) Heh, just tried myself, I guess not; still got disconnected after 15 minutes. [17:25:14] 10Analytics, 10EventStreams: EventStreams socket stays connected without any traffic incoming - https://phabricator.wikimedia.org/T250912 (10Ottomata) Hm, this is going to be hard to reproduce, especially given T179986. I'd expect this to be a client side problem, but it could be something on EventStreams sid... [17:28:55] hmm moritzm i still get [17:29:10] broken apt-get update on e.g. kafka-jumbo1007 [17:29:21] buster-wikimedia InRelease' doesn't have the component 'thirdparty/confluent' [17:29:27] i've run puppet on apt.w [17:29:40] apt1001 [17:31:53] hm yeah it isn't there i guess [17:31:53] https://apt.wikimedia.org/wikimedia/dists/buster-wikimedia/thirdparty/ [17:33:04] thirdparty/confluent is listed correctly in buster-wikimedia distributions [17:35:07] hmm i think reprepro update isn't working [17:35:12] beacuse of rocm repo? [17:35:13] elukey: yt? [17:35:25] oh no you are off [17:35:26] sorry [17:58:58] ah i think i sudoed incorrectly [18:04:06] hey a-team: a data source question. The AQS mediarequests API, is that the same data as wmf.mediarequests in the Data Lake? [18:04:43] wmf.mediarequest, sorry, it's singular [18:05:23] from what I know, the Search Team is interested in ingesting it into ElasticSearch [18:11:39] Nettrom: yea the data in the apis comes from wmf.mediarequest and wmf.mediacounts (before May 2019 iirc) [18:12:00] ottomata: o/ [18:12:07] I am here if you need a brain bounce [18:12:30] ah okok already solved, goooood [18:12:40] fdans: excellent, thanks so much! [18:12:47] * elukey afk again :) [18:19:17] Nettrom: as far as we know there is no ingestion of mediarequest data into elastic but gehel might know whether they have a project scheduled for that [18:38:18] nuria / Nettrom : we are planning on experimenting with ingesting search requests into an elasticsearch cluster [18:38:39] gehel: just asked erik on #wikimedia-discovery [18:38:42] But this is at experimentation phase only [18:59:08] nuria: and it looks like I completely misunderstood the context :) [18:59:18] * gehel should not reply to IRC while having dinner :/ [19:19:07] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10nshahquinn-wmf) The Legal team looked into this and said they do not see any legal impediment to either choice, as long as we are clear and consistent about it. It's also worth noting that... [19:20:36] 10Analytics, 10Event-Platform: DRY kafka broker declaration into helmfiles from puppet - https://phabricator.wikimedia.org/T253058 (10Ottomata) [19:32:08] 10Analytics, 10Product-Analytics: [Spike] Should EventLogging support DNT? - https://phabricator.wikimedia.org/T252438 (10mforns) > It's also worth noting that our privacy policy currently states that we do not respond to the Do Not Track header. I retract my previous comment, then. I'd be OK with ignoring DNT... [19:44:47] 10Analytics, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T252915 (10Milimetric) Yeah, the April job failed, I'm not exactly sure why, so I'm erasing the April line and letting it rerun. It should be fixed in a coupl... [19:52:15] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10EBernhardson) Not sure if it's related, but I've noticed one of my data pipelines that n... [20:22:13] 10Analytics, 10Analytics-Kanban, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T252915 (10Milimetric) p:05Triage→03Medium Ok, fixed, it will take a little while to sync to the public files, but it looks good. St... [20:41:41] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform - https://phabricator.wikimedia.org/T249261 (10Ottomata) Hm, not related to this ticket, but possibly to T252203. Looking [20:55:50] 10Analytics, 10Analytics-Kanban: Camus failing to import eqiad.mediawiki.(api|cirrussearch)-request from partitions leaders on kafka-jumbo1006 - https://phabricator.wikimedia.org/T252203 (10Ottomata) In https://phabricator.wikimedia.org/T249261#6146372 @EBernhardson reported that he was missing `event.mediawik... [21:05:31] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 2 others: Set up an instance of EventStreams in beta that will allow for consuming any stream - https://phabricator.wikimedia.org/T253069 (10Ottomata)