[00:13:32] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Neil_P._Quinn_WMF) [00:13:38] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Neil_P._Quinn_WMF) [00:25:09] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) a:03awight [00:27:58] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10Neil_P._Quinn_WMF) p:05Triage→03High [00:49:46] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10Nuria) @Neil_P._Quinn_WMF Per my conversation with @kzimmerman seems like the mo... [01:01:24] 10Analytics, 10Readers-Web-Backlog: [Bug] Many JSON decode ReadingDepth schema errors from wikiyy - https://phabricator.wikimedia.org/T212330 (10Jdlrobson) [02:06:03] 10Analytics: Clean up home dirs for users jamesur and nithum - https://phabricator.wikimedia.org/T212127 (10Jalexander) >>! In T212127#4831160, @fdans wrote: > Stuff to delete in both users > > jamesur: > - home dir in stat1005 > - home dir in hdfs > - database `jamesur` in hive (1 table) If you haven't alr... [03:12:19] 10Analytics, 10Research: Generate article recommendations in Hadoop for use in production - https://phabricator.wikimedia.org/T210844 (10leila) [06:34:22] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) the views in question here could be created a little faster than monthly. But still, even every two... [06:39:36] 10Analytics, 10Analytics-EventLogging: eventlogging fails flake8 due to new upstream version, breaking CI - https://phabricator.wikimedia.org/T212396 (10Legoktm) p:05Triage→03High [06:40:02] 10Analytics, 10Analytics-EventLogging: eventlogging fails flake8 due to new upstream version, breaking CI - https://phabricator.wikimedia.org/T212396 (10Legoktm) [07:45:36] hello :) [07:45:48] I am rolling restart druid middlemanagers for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480733/ [08:05:40] done [08:31:08] 10Analytics: Clean up home dirs for users jamesur and nithum - https://phabricator.wikimedia.org/T212127 (10elukey) @Jalexander done! There is now a directory on stat1007 (stat1005 is deprecated) called `jamesur` in `foks`'s home directory (owned by `foks:root and read/write/execute only for him). Is there anyth... [08:51:58] joal: o/ [08:52:10] if you have time I'd add the two journal nodes [10:26:44] joal elukey hellooo, I'm getting this when running a query in mediawiki_history_reduced in hive [10:26:51] Container [pid=16496,containerID=container_e98_1544022186674_62202_01_001048] is running beyond physical memory limits. Current usage: 2.9 GB of 2 GB physical memory used; 4.7 GB of 4.2 GB virtual memory used. Killing container. [10:27:40] don't know if it's anything worth looking into [10:27:48] well how big is this query? [10:27:58] it seems that it breaches the Xmx of the jvm [10:28:21] I think that you can override the setting for a specific job [10:28:32] but I have no idea how [10:29:47] fdans: can you try with something like https://community.hortonworks.com/content/supportkb/48788/i-am-seeing-outofmemory-errors-when-i-run-a-hive-q.html ? [10:29:54] maybe not those huge values [10:29:58] something like 4096 [10:57:01] fdans: ? [10:57:19] elukey: oh sorry luca thought I had replied [10:57:34] in the end I found the info I needed with a different query, no need to change limits [10:58:12] super :) [11:24:32] 10Analytics, 10Analytics-Kanban, 10WMDE-Analytics-Engineering, 10Wikidata, and 3 others: track number of editors from other Wikimedia projects who also edit on Wikidata over time - https://phabricator.wikimedia.org/T193641 (10Addshore) I don't remember when this is set to run @JAllemandou , but I see no da... [11:25:47] * elukey lunch + errand! [12:01:46] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): log-events topic emitted in EventBus - https://phabricator.wikimedia.org/T155804 (10mobrovac) [12:03:14] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 2 others: Git Commit hook that adds a whole new file when a new version of schema is committed - https://phabricator.wikimedia.org/T206812 (10mobrovac) [12:04:37] 10Analytics, 10EventBus, 10Operations, 10Core Platform Team Backlog (Watching / External), 10Services (watching): eventbus should send statsd in batches - https://phabricator.wikimedia.org/T141524 (10mobrovac) [12:06:45] 10Analytics, 10ChangeProp, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later): RESTBase content rerenders sometimes don't pick up the newest changes - https://phabricator.wikimedia.org/T176412 (10mobrovac) [12:06:49] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Later), 10MW-1.32-notes (WMF-deploy-2018-09-25 (1.32.0-wmf.23)), and 4 others: Convert all hooks to EventFactory - https://phabricator.wikimedia.org/T204575 (10mobrovac) [12:07:01] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later), and 2 others: EventBus should make better use of DI - https://phabricator.wikimedia.org/T204295 (10mobrovac) [12:07:08] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later), and 2 others: EventBus should not use service container in application logic - https://phabricator.wikimedia.org/T204296 (10mobrovac) [12:08:23] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later): Reliable (atomic) MediaWiki event production - https://phabricator.wikimedia.org/T120242 (10mobrovac) [12:09:07] 10Analytics, 10ChangeProp, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later): Support per-topic configuration in EventBus service - https://phabricator.wikimedia.org/T157092 (10mobrovac) [12:16:48] 10Analytics, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (next), 10Wikimedia-production-error: Timeouts on event delivery to EventBus - https://phabricator.wikimedia.org/T180017 (10mobrovac) [12:17:25] 10Analytics, 10ChangeProp, 10EventBus, 10Core Platform Team Backlog (Later), and 2 others: Enable controlled debug logging for change-prop - https://phabricator.wikimedia.org/T189621 (10mobrovac) [12:17:43] 10Analytics: Clean up home dirs for users jamesur and nithum - https://phabricator.wikimedia.org/T212127 (10Jalexander) >>! In T212127#4836777, @elukey wrote: > @Jalexander done! There is now a directory on stat1007 (stat1005 is deprecated) called `jamesur` in `foks`'s home directory (owned by `foks:root and rea... [12:22:36] Hi elukey - Happy to help with journalnodes when you're back :) [12:23:28] 10Analytics, 10Analytics-Kanban, 10WMDE-Analytics-Engineering, 10Wikidata, and 3 others: track number of editors from other Wikimedia projects who also edit on Wikidata over time - https://phabricator.wikimedia.org/T193641 (10JAllemandou) Same exact problem as last month: job has run, but no data is presen... [12:27:22] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10JAllemandou) The query you bookmarked is doing what you expect I guess :) The next steps are to create the associated table and create new partitions based on the query.... [12:35:04] fdans: The issue you have experienced with querying mediawiki-history is due to default mapper RAM-size behing too small to correctly handle the parquet files - The solution is to grow the size of mappers-memory - In hive: `set mapreduce.map.memory.mb=4096; set mapreduce.map.java.opts=-Xmx3276m;` [12:42:02] (03PS3) 10Joal: Update mediawiki-history comment and actor joins [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/480796 (https://phabricator.wikimedia.org/T210543) [12:42:55] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Core Platform Team Kanban (Doing), 10Services (doing): Create scripts to estimate Kafka queue size per wiki - https://phabricator.wikimedia.org/T182259 (10mobrovac) [12:43:11] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Generalized rate limiting, deduplication, and job scheduling module - https://phabricator.wikimedia.org/T173447 (10mobrovac) [12:44:05] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088 (10mobrovac) [12:47:08] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Make Kafka JobQueue use Special:RunSingleJob - https://phabricator.wikimedia.org/T182372 (10mobrovac) [12:47:16] 10Analytics, 10MediaWiki-API, 10RESTBase-API, 10Core Platform Team Kanban (Blocked Externally), 10Services (blocked): Top API user agents stats - https://phabricator.wikimedia.org/T142139 (10mobrovac) [12:47:22] 10Analytics, 10RESTBase, 10Core Platform Team Kanban (Blocked Externally), 10Services (blocked): REST API entry point web request statistics at the Varnish level - https://phabricator.wikimedia.org/T122245 (10mobrovac) [12:54:23] 10Analytics, 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Core Platform Team Backlog (Designing), 10Services (designing): ORES hook integration with EventBus - https://phabricator.wikimedia.org/T201869 (10mobrovac) [12:55:14] 10Analytics, 10ChangeProp, 10EventBus, 10WMF-JobQueue, and 2 others: Consider disabling automatic topic creation in main-kafka - https://phabricator.wikimedia.org/T199432 (10mobrovac) [12:56:29] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Allow easy tuning of the jobqueue concurrency. - https://phabricator.wikimedia.org/T175800 (10mobrovac) [13:05:48] (03PS1) 10Joal: Add druid-kafka task example in banner_activity [analytics/refinery] - 10https://gerrit.wikimedia.org/r/480956 (https://phabricator.wikimedia.org/T203669) [13:07:33] 10Analytics, 10Analytics-Kanban, 10Fundraising-Backlog, 10Patch-For-Review, 10User-Elukey: Return to real time banner impressions in Druid - https://phabricator.wikimedia.org/T203669 (10JAllemandou) Druid-kafka-supervisor task and how-to added to refinery in `oozie/banner_activity/druid` folder (https://... [13:40:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Prototype in node intake service - https://phabricator.wikimedia.org/T206815 (10CCicalese_WMF) [13:59:26] joal: o/ [13:59:32] Heya [14:00:01] going to merge a couple of things, then journal nodes? [14:00:18] elukey: Will drop at 16:00 for kids, can be before? [14:00:37] of course, in 10m I'll be ready [14:00:53] Awesome - thanks elukey :) [14:14:51] (03CR) 10Elukey: [C: 03+1] "One nit but looks good!" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/480956 (https://phabricator.wikimedia.org/T203669) (owner: 10Joal) [14:17:48] joal: all right almost ready to go [14:17:57] \o/ :) [14:18:03] going to follow https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration#Adding_a_new_JournalNode_to_a_running_HA_Hadoop_Cluster [14:19:15] elukey: do you want us to batcave for me to follow? [14:19:51] sure [14:34:08] o/ elukey sorry i didn't get to look at superset yesterday [14:34:11] will do now [14:36:46] ottomata: np! I am expanding the journals now with joal [14:37:48] ok great!~ [14:53:14] elukey: I tried to catch up with you on security review tasks :) I've been in a deep dark hole with other thigns for what seems like years...probably just weeks tho. stupid q unrelated, what is weblog1001 used for? sampled webrequest logs land there right? But this is all dupe data from hadoop? Is it some kind of alerting or quicky analytics? [14:57:12] chasemp: its because ops likes it [14:57:16] we dont' use it :) [14:57:29] they like to grep/awk/sed on files to find quick issues [14:58:35] gotcha tx [15:00:16] chasemp: o/ [15:00:28] yeah techically that thing is not analytics :) [15:04:47] 10Analytics, 10Phabricator, 10Wikimedia-Stream: Move KafkaSSE development from Differential to Gerrit - https://phabricator.wikimedia.org/T212420 (10Ottomata) p:05Triage→03Normal a:03Ottomata [15:05:07] 10Analytics, 10Operations, 10Performance-Team, 10Traffic: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) Plain nginx config has the ability to remove the headers, but it can't do so conditionally... [15:08:50] ottomata: we have 7 journal nodes now \o/ [15:08:51] all good [15:09:03] do you think that we could merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480965/ after grooming? [15:09:09] basically doing the same procedure [15:09:22] but stopping the journal nodes instead of creating new ones [15:09:36] woohoo [15:09:47] elukey: sure thang [15:11:49] nice :) [15:24:38] 10Analytics, 10Operations, 10Performance-Team, 10Traffic: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) @BBlack would you miss x-cache, x-cache-status and x-varnish if those were completely removed at the TLS termination level? Some o... [15:26:48] ah elukey,k not 100% sure yet, but i think your superset dependency version problems are fine. [15:26:51] i'm looking just at markdown [15:27:01] the version we have in wikimedia branch doesn't specify a version for makrdown [15:27:15] is there another one maybe? [15:27:38] flask-appbuilder ok that one has changed [15:27:53] and I see it get the wikimedia branch version. [15:32:12] ottomata: https://github.com/wikimedia/incubator-superset/blob/wikimedia/requirements.txt#L21 ? [15:33:14] ? hm [15:33:21] oh i was looking at setup.py [15:33:42] ah ok! [15:33:44] hm [15:34:23] hey luca, sorryyyyy [15:34:27] omw [15:34:58] elukey: i think pip is using the setup.py stuff [15:35:17] "Whereas install_requires metadata is automatically analyzed by pip during an install, requirements files are not, and only are used when a user specifically installs them using pip install -r. [15:35:17] " [15:35:37] i think its ok [15:35:58] gonna make a commit for this. i made an small improvement to build_wheels.sh [15:37:01] ah but the npm stuff [15:37:01] ottomata: mmm I don't get it, why when I build the wheel manually it does pick up the right deps? [15:37:02] not sure about that yet [15:37:06] it is not ok in this case [15:37:13] elukey: how are you building the wheel manually? [15:37:36] I added the procedure in the Readme the last time [15:37:46] there is a command for setuptools [15:37:51] don't recall it exactly [15:37:57] it creates the tar.gz and the wheel [15:38:01] under dist [15:38:06] oh looking [15:38:26] the main problem is that markdown breaks 0.26.3 if used [15:38:29] ottomata, you think this would work as a time condition given since and until for Hive to prune partitions? https://pastebin.com/5Axxiknq [15:38:32] they fixed weird bugs afterwards [15:38:34] 10Analytics, 10Operations, 10Performance-Team, 10Traffic, 10Patch-For-Review: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10BBlack) I don't know off-hand if we can live without them all for manual debugging and such, or if nginx is the best... [15:38:49] ottomata, I'll brb for 20 mins [15:39:11] ottomata: python3 setup.py sdist bdist_wheel [15:39:17] mforns_brb: maybe, idea though: what about just examining the table partition fields? [15:39:19] https://gerrit.wikimedia.org/r/#/c/analytics/superset/deploy/+/479249/4/README.md [15:39:19] i think that should be possible [15:39:37] ya looking [15:39:38] 10Analytics, 10Operations, 10Performance-Team, 10Traffic, 10Patch-For-Review: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) Is there an nginx "site" or config specific to varnish termination? [15:39:52] ottomata: but I haven't tried it with this use case though, now that I think about it [15:39:56] I tried it with 0.28.x [15:40:01] so it might be weird as well [15:40:03] uff [15:40:19] elukey: why sdist? [15:40:19] I can try on builder now [15:40:36] can be avoided yes [15:40:53] I think I picked it up from some people reccomending it on the internetz and didn't remove it [15:41:01] the important one was bdist_wheel IIRC [15:41:09] ottomata: not sure if you saw adam's git fat problem above, but I agree with him, I get the same error when I do git fat pull: [15:41:18] ERROR:git-fat: Error reading or parsing configfile: /home/milimetric/projects/refinery/.gitfat [15:42:00] it doesn't really stop me from doing most things, but we should probably remove that from the docs or fix it [15:42:06] ? [15:42:09] it seems to work for me [15:42:22] :) that doesn't help [15:42:33] you've done git fat initi ? [15:42:38] yes, that works fine [15:42:45] I have version 0.5.0 like Adam [15:42:48] weird [15:42:53] what does your .gitfat file look like? [15:42:59] it happened before, I think you told me to ignore it [15:43:12] oh, yeah i mean unless you need all the artifact jars locally [15:43:14] https://www.irccloud.com/pastebin/TGYma2Wy/ [15:43:14] 10Analytics, 10Operations, 10Performance-Team, 10Traffic, 10Patch-For-Review: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) Could be a puppet variable too, to make the filtering block conditional. [15:43:34] milimetric: maybe bad version of git fat? or maybe I have an old version [15:43:38] will look into it... [15:43:41] yeah, I don't need them, but it's in the readme that you should do this to set up the repo, so we should fix this one way or another [15:44:34] aye [15:44:44] let's just add a conditional to the readme. [15:44:48] IF you want them, then do this... [15:45:50] but someone new to refinery wouldn't know if they want them, and if they do want them it would still be broken [15:46:04] what version of git fat do you have? [15:46:23] ottomata: one qs - why would the superset people have requirements.txt super precise and setup.py not? [15:46:30] I mean it is really strang [15:46:32] *strnage [15:46:35] ufff [15:46:45] elukey: some people pull requirements.txt from setup, is that what's happening? [15:47:35] milimetric: not really, they have install_requires in setup.py that differs from requiremnets.txt [15:47:53] elukey: mind pointing me to the code/branch you're looking at? [15:48:49] https://github.com/wikimedia/incubator-superset [15:48:54] milimetric: sure [15:48:55] so https://github.com/wikimedia/incubator-superset/blob/wikimedia/setup.py#L79 [15:48:59] vs [15:49:00] milimetric: [15:49:00] https://github.com/wikimedia/incubator-superset/tree/wikimedia [15:49:01] sorry that one [15:49:02] yeah [15:49:09] https://github.com/wikimedia/incubator-superset/blob/wikimedia/requirements.txt#L21 [15:52:13] 10Analytics, 10Operations, 10Performance-Team, 10Traffic, 10Patch-For-Review: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10BBlack) `localssl.erb` would probably be more appropriate and is the site file, but it's a generic TLS reverse proxy... [15:55:27] ok, elukey I have an answer for you but it's ugly [15:55:29] "If the pull request adds a Python dependency include it in setup.py denoting any specific restrictions and in requirements.txt pinned to a specific version which ensures that the application build is deterministic." [15:56:05] which, translated, means "we don't understand how python works and we're just going to do a bunch of random shit and pretend like we're cool" [15:56:07] ottomata: confirmed that my build doesn't work either, sorry :( [15:56:40] seriously, this is one of the ugliest python repos I've seen in a while [15:57:06] milimetric: so in theory requirements.txt and setup should be aligned? [15:57:32] usually they use one or the other [15:57:32] not both [15:57:39] elukey: what everyone else does is put everything in requirements.txt and read them from setup.py [15:57:50] sigh [15:57:54] these people duplicate it not just there but also in requirements-dev.txt, which WTF [15:58:03] elukey: yours with sdist etc. doesn't build with the right deps? [15:58:11] ottomata: nope, same issue [15:58:13] just tried [15:58:24] it pulls setup.py's ones as you were saying [15:58:30] ok, then maybe we will just patch our branch to make setup.py to use requirements.txt ... :p [15:58:42] cool, i was about to try that :) [15:58:42] elukey: but you can build with just the requirements.txt reqs [15:59:02] pip install -r requirements.txt [15:59:03] ya milimetric but we need to pip to do that as part of build process [15:59:09] we were hoping not to have to clone locally [15:59:18] and instead point at github git url [16:00:08] I'm not sure I understand our fork and how this is set up, but happy to chat about it if you're stuck [16:00:29] but elukey i think you are right, we will need to probably clone and build anyway [16:00:35] since we do need to run the webpack npm stuff [16:00:49] milimetric: previously we just pointed at a superset pypi version [16:00:52] we can't do taht anymore [16:00:52] so [16:01:07] we changed our superset-deploy repo frozen-requirement.txt [16:01:15] to instead of saying superset==x.x.x [16:01:16] to [16:01:17] something like [16:01:30] ok, gotcha, then yeah, do the requirements pull thing from setup, one sec [16:01:32] # Wikimedia uses a fork of superset to apply our own fixes. [16:01:32] git+https://github.com/wikimedia/incubator-superset.git@wikimedia#egg=superset [16:01:49] but, using that for pip install causes it to build with setup.py requirements [16:02:02] whereas apparently the pypi ones were built with requirements.txt [16:02:20] but anyway, we are going to have to manually build isntead of let pip do it from URL, because there is an extra step to webpack the JS stuff [16:02:54] i think we can script this up as part of build though luca... [16:02:56] working on it [16:03:15] 10Analytics, 10Operations, 10Performance-Team, 10Traffic, 10Patch-For-Review: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) I wasn't aware that the latest plan was to use ATS for TLS termination. There might be a way to do this in... [16:03:22] https://github.com/apache/incubator-superset/issues/5527#issuecomment-430082919 :P [16:03:25] milimetric: --^ [16:03:32] https://github.com/wikimedia/analytics-wikimetrics/blob/master/setup.py#L25 [16:03:35] elukey: ^ [16:04:47] milimetric: neat! [16:04:52] elukey: lol, I'm really curious to know what source you would read this kind of advice in: https://github.com/apache/incubator-superset/issues/5527#issuecomment-430738065 [16:05:33] elukey: yeah, and that's a project we haven't touched in like 3 years... superset is seen by thousands of people... I really am at a loss here [16:05:46] cool [16:06:31] milimetric: is it possible to make setup.py run a script on dist build? [16:06:46] i'm thikning we might be able to just patch our fork setup.py [16:06:49] and not have to do manual clone stuff [16:08:45] ottomata: that's what I linked to in wikimetrics, it just parses the requirements.txt file [16:08:54] (03CR) 10Nuria: Add druid-kafka task example in banner_activity (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/480956 (https://phabricator.wikimedia.org/T203669) (owner: 10Joal) [16:09:03] milimetric: no [16:09:06] there are two issues [16:09:09] sorry I'm slow today, everything's going in slow motion after last night [16:09:21] the dependency requirements thing, which your link will solve [16:09:21] and [16:09:33] https://github.com/wikimedia/incubator-superset/blob/master/pypi_push.sh [16:09:38] this is what they run when they release [16:10:01] we need to run npm run build [16:10:04] to run webpack for the dist [16:10:14] we can do it if we clone locally and then build manually in a scrip [16:10:27] but i'd prefer if i could just link to the fork in our frozen-requirements.txt [16:10:45] and then have our forked setup.py automatically run the npm run build when being installed [16:11:57] ottomata: you're not trying to publish anything to pypi, right, you're just saying we also need to build npm? [16:12:08] *build with npm? [16:12:36] right [16:12:45] its to webpack the statick stuff i think [16:12:50] maybe it install node packages too [16:13:02] NODE_ENV=production webpack --colors --progress [16:14:38] am googling, will find [16:14:47] ottomata: https://jichu4n.com/posts/how-to-add-custom-build-steps-and-commands-to-setuppy/ [16:15:07] that has a nice walkthrough [16:15:14] nice, found simliar things but not this one [16:15:26] look at the second part, where he adds it to setup.py build [16:15:51] the magic word is build_py in that cmdclass object [16:16:13] if it works let's try also to work with upstream as much as possible [16:16:25] ideally it would be great to get these things fixed on their side [16:17:13] ya ok, i will commit to a second branch in our repo (upstream-fixes or something) this change, then we can merge that into wikimedia branch and build from there. and we can try to PR from upstream-fixes [16:18:18] super thanks :) [16:21:20] ottomata: anything against me trying to remove two journal nodes? [16:27:30] elukey: not at all proceed! [16:27:33] i'm here if you need me [16:27:44] ack! [16:47:22] elukey: how did you run pypi_push.sh on builder? [16:47:26] i don't have webpack installed [16:47:32] did you do an npm install somewhere? [16:47:40] yeah on the host [16:47:42] ah [16:47:50] so you did cd assets/dist [16:47:52] npm install -g [16:47:55] or something [16:47:55] ? [16:47:59] yep [16:48:02] k [16:50:18] elukey: did you have this problem? [16:50:19] https://github.com/wikimedia/incubator-superset/blob/wikimedia/superset/assets/package.json#L44 [16:50:24] npm ERR! Invalid package name "//": name can only contain URL-friendly characters [16:52:52] I didn't with 0.28 but I did earlier on when I tried to build, I thought it was a temporary glitch and didn't check further (I only neeed the wheel for the deps) [16:53:06] tried to build 0.26.3 [16:53:12] right, but the npm run build won't work with this package.json it seems [16:53:13] oh [16:53:14] i see [16:54:03] hm no [16:54:06] i'm on 0.26.3 [16:54:10] with frans patch [16:54:12] the wikimdia branch [16:54:15] it has the "//" [16:54:33] https://github.com/wikimedia/incubator-superset/blob/wikimedia/superset/assets/package.json#L44 [16:54:49] it looks like those are supposed to be comments? [16:56:24] (03CR) 10Joal: Add druid-kafka task example in banner_activity (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/480956 (https://phabricator.wikimedia.org/T203669) (owner: 10Joal) [16:56:34] (03PS2) 10Joal: Add direct kafka-to-druid ingestion example [analytics/refinery] - 10https://gerrit.wikimedia.org/r/480956 (https://phabricator.wikimedia.org/T203669) [16:58:38] PROBLEM - Hadoop JournalNode on analytics1028 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.qjournal.server.JournalNode [16:58:44] PROBLEM - Hadoop JournalNode on analytics1035 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.qjournal.server.JournalNode [16:59:35] hehe --^ [16:59:55] wooo sorry team forgot to email, will miss today's meetings I'm omw to pick my parents up from the station [17:00:01] a-team * [17:01:05] joal: something didn't go well at the last step, namely failing over back to an-master1001 [17:01:12] :( [17:01:20] PROBLEM - HDFS missing blocks on an-master1001 is CRITICAL: 4879 ge 5 https://grafana.wikimedia.org/dashboard/db/hadoop?var-hadoop_cluster=analytics-hadooporgId=1panelId=40fullscreen [17:01:24] elukey: obviously the step at which I needed to leave :( [17:01:38] elukey: any idea about the reason? [17:01:46] no idea [17:02:53] :S [17:05:40] not sure why it reports missing blovks [17:05:42] *blocks [17:05:53] really everything was fine when 1002 was leader [17:10:21] elukey: huge GC times lately on an-master1001 :( [17:13:21] I am running fsck / to see what comes up, for the moment only under replicated blocks reported [17:13:50] nope failed half way through [17:14:17] :S [17:14:55] elukey: grafana reports underreplicated blocks, pending replication blocks, and under-replcated blocks as well [17:14:57] so I suspect that an1001 is lagging terribly in bootstrapping properly [17:15:05] elukey: very possible [17:15:28] elukey: or, it hasn't liked the removal of the 2 old journalnodes? [17:17:03] so in theory this was a simpler op that before, since I manually stopped them [17:17:12] and just reduced the number of them [17:17:35] right - hm - have they removed from the namenode conf? [17:17:49] the major issue now is that some blocks are not found by the hdfs namenode, and those are reported by the datanodes when requested [17:17:57] yep yep [17:18:02] weird [17:18:43] so one possible action could be to failover to an1002 and see if things are better in there [17:19:25] elukey: I +1 this idea, between fearing good or worse :) [17:20:01] I am wondering if the namenode is now in a state in which it has a partial view of the blocks, and it is trying to replicate them as well [17:20:09] elukey: very possible [17:20:35] elukey: number of pending-rep blocks moves while under and missing are not moving - seems related to metadata correctness [17:21:27] elukey: I feel NN is trying to rebuild a correct view of where blocks are and has issues with it [17:21:53] yep [17:26:47] a-team; canceling groskin [17:26:52] k nu [17:29:44] ottomata, I understand your suggestion (re. partition pruning), I just thought that we would do that outside of that function, and pass to it just that field, it might be that you already know which is the smallestPartition for a given Spark job, and you don't need to inspect the partitions. [17:31:17] ottomata: can you join batcave-2? [17:36:29] RECOVERY - HDFS missing blocks on an-master1001 is OK: (C)5 ge (W)2 ge 0 https://grafana.wikimedia.org/dashboard/db/hadoop?var-hadoop_cluster=analytics-hadooporgId=1panelId=40fullscreen [17:38:35] mforns: https://gist.github.com/ottomata/9abaf0cefcdd9af9bcaba36c8ec3968d [17:38:36] elukey: coming [17:52:04] mforns: let's discuss [17:52:22] ottomata, ok [17:54:15] ottomata, you can also pass the list of partitions to the function, and then instead of: if (smallestPartition == "hour"), you would do: if (partitions.contains("hour")) [17:55:20] mforns: but if you know the partitions keys [17:55:27] we can assume that: if hour is there, then day is there. if day is there, month is there. If month is there, year is there. [17:55:28] then you can just extrapoloate from your timestamps [17:55:43] ottomata, how? [17:56:17] minTimestamp = 2018-12-01T05:00:00 [17:56:29] partitionKeys = year,month,day,hour [17:57:29] parititonClause = "" [17:57:29] for k in parittionKeys: [17:57:29] partitionClause += minTimestamp.get(k) [17:57:32] sory [17:57:46] probably build array of year=YYYY [17:57:48] and htne join with AND [17:58:04] so you'd end up extracting each of the partition values from the timestamp [17:58:09] into an array like [17:58:16] ottomata, I see, but that only works if you're trying to load just 1 partition [17:58:29] ["year=2018", "month=12", "day=01", "hour=05"] [17:58:42] ohhhhhh beacuse you need a between right [17:59:00] right right right...this is why milimetric wants us to switch to dt= partitions everywhere [17:59:06] if you want to load since 7 days ago until 2 days ago?> [17:59:40] :) [17:59:43] hehe [18:00:03] yes dt partitions would make this way easier, but I think for now the expression works [18:00:38] hm mforns your thing would still work with partition keys, no? [18:00:41] if you had the keys [18:00:45] ottomata, yes [18:00:47] then you just change your cases [18:00:53] exactly [18:01:10] i think that'd be nicer than providing mostGranularTimePartition, because then the user doesn't have to think about it [18:01:12] we can assume that if hour is there, then day is there; and if day is there means month is there [18:01:13] they jsut provide the timestamps [18:01:13] etc [18:01:16] right [18:01:31] year should always be there at least [18:02:11] milimetric: do we have dt= partition in use anywhere now? i forget? [18:02:17] ok, will change it [18:02:30] ottomata: yes, interlanguage table [18:02:46] https://github.com/wikimedia/analytics-refinery/blob/master/hive/interlanguage/create_interlanguage_navigation_table.hql#L18 [18:03:14] we should really just make a nice UDF like between_dates [18:03:24] mforns: maybe you can add a case for that? [18:03:34] a UDF would work, then spark could just use it when getting the df [18:03:48] mforns: adding the date use case not necessary, but might be nice eventaully [18:03:49] ottomata, milimetric, I could do a UDF [18:03:50] doesn't have to be this patch [18:04:00] a UDF would be nice for others using hive [18:04:08] and spark could use it [18:04:12] aha [18:04:39] i think anyway... :) [18:04:41] we said we would a long time ago [18:04:53] but I forget why we didn't do it [18:05:07] mforns: up to you here, that might be a lot more work [18:05:14] adding it as spark function for now would be fine [18:06:36] I think I can not possibly finish this now, until tomorrow (end of quarter) anyway, so it might be a good moment to start the UDF dunno? nuria? [18:07:20] ottomata, but you're sure SparkSQL can use the UDFS? [18:07:25] not100% sure :) [18:07:30] but i think so [18:07:32] should check [18:07:36] ottomata, mforns to avoid scope creep let's do solution in spark now and wrap up this work [18:07:37] ok, can check [18:07:48] ok, right spark first [18:07:58] ottomata: if you ok I'd failover to 1001, seems stable [18:08:25] but I am wondering if we should instead bump the heap to 12G first [18:08:45] ottomata, nuria, should I put this funtion somewhere it can be reused (like ./refinery-spark/src/main/scala/org/wikimedia/analytics/refinery/spark/utils) or just as part of HiveToDruid for now? [18:09:21] ottomata, mforns : and write ticket for udf if we know it can be used by spark sql which seems like it shoudl be possible [18:09:34] k [18:10:37] https://stackoverflow.com/questions/40369170/registering-hive-custom-udf-with-spark-spark-sql-2-0-0 [18:21:41] elukey: why not let's bump heap first [18:21:51] mforns_brb: spark/utils is good [18:22:15] OOO elukey [18:22:17] sorry [18:22:22] OO mforns [18:22:33] you could put it on the HiveExtensions DataFrame thing... :D [18:22:39] then you can call it on the df itself [18:23:10] we'd add two new function sthere [18:23:17] df.hivePartitionColumnNames [18:23:18] and [18:23:40] df.partitionBetweenWhereClause [18:23:42] (or whatever) [18:23:48] somethign like that [18:24:16] mforns_brb: down near the bottom of the HiveExtensions.scala file [18:24:23] is implicit class DataFrameExtensions(df: DataFrame) [18:24:29] OHhhhh but hm [18:24:40] ottomata: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/481011/ - ok? [18:24:41] oh wait, you are trying to get this info before you create a df [18:24:41] hmmm [18:24:52] mayyybe not worth it then [18:25:11] +1 elukey [18:25:20] super thanks, merging and restarting the namenode on 1001 [18:25:45] k [18:28:20] ah also going to remove the journalnode daemons package etc.. from 1028/1035 [18:28:32] ottomata: on that patch is our initial heap same than our final heap? [18:28:35] cc elukey [18:28:48] nuria: yep [18:28:53] elukey: +1 [18:29:41] elukey: is it always been that way? [18:30:15] nuria: ya, we do that mainly so we don't have to worry about trying to grab stuff later. we want the process to just reserve what it might ever try to and use it. [18:34:31] then new namenode-12g-monster is working fine :D [18:37:52] ok journal nodes cleaned up [18:37:58] (1028-35, the old ones) [18:39:19] gc old gen time looks very good (~3/400 ms compared to 4/5s) [18:41:44] ottomata: ready to failover if you are ok [18:52:51] ok elukey go for it! [18:53:55] done and it worked in a second [18:54:03] I don't see any GC for old gen anymore [18:54:05] wooooowwwwww [18:54:08] \o/ [18:54:22] going to wait 5 mins and then I'll restart 1002 to apply heap settings [18:54:33] great! [18:55:17] elukey: i don't think we'll be able to upstream these superset changes...i'm second guessing the idea to do it in the setup.py file [18:55:30] :( [18:56:05] i'm having trouble here and there shelling out to e.g. npm run build, not entirely sure why [18:56:10] but it works directly on the CLI [18:56:22] actually no, it doesn't, npm run build doesn't, but webpack directly does [18:56:41] hmm, actually if i can get this to work, we should use it, but I think not try to upstream [18:57:02] agreed, eventually we'll switch back to their build (hopefully) [19:04:22] loading fsimage during safe mode [19:04:25] 1 mins, 12 sec with 12G [19:04:30] 3/4 mins with 8G [19:04:31] lol [19:04:48] nice! [19:05:11] already bootstrapped, it takes a couple of minutes now [19:05:13] really gooooood [19:10:09] Just back [19:10:16] Looke like nanode is happy :) [19:10:20] Thanks a elukey :) [19:10:25] +lot [19:10:38] \o/ [19:10:43] did you see GC old gen time? [19:11:20] it is 0 now [19:11:24] * elukey dances [19:11:38] hehe :) [19:11:45] With a bump of Heap ;) [19:12:35] all right going to dinner! [19:12:42] have a good afternoon/evening folks [19:12:44] ping me if needed [19:12:45] Enjoy elukey - See you tomorrow [19:12:50] very nice!~ [19:12:51] laters!~ [19:12:53] o/ [19:13:10] looks like ottomata fields ~y [19:13:19] Should I say tildy? [19:15:22] ok ottomata will see what I can do thanks! [19:18:53] (03PS1) 10Awight: Schema for ORES scores [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) [19:19:22] haha tildy is close to exclamationy and i am sloppy [19:24:01] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current): Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) With the schema above, I was able to insert rows using this query: ` lang=sql set hive.exec.dynamic.partition.mode = 'nonstrict'; insert i... [19:24:17] (03CR) 10Awight: [C: 04-1] "WIP'ing to fix a bug in the "errors" column." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [19:27:26] (03CR) 10Ottomata: "Some nits :) This will be useful, thanks!" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [19:30:09] Hi nuria - Are you nearby? [19:34:50] (03CR) 10Awight: "Unblocking on "errors" because I don't see a convenient way to filter out just the errors on one model. It's fine to have redundant error" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [19:35:58] (03PS2) 10Awight: Schema for ORES scores [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) [19:38:49] (03CR) 10Ottomata: "Re errors, hm. Maybe you don't want the errors at all? Could you just exclude any records from mediawikI_revision_score with errors and " [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [19:39:55] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current): Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) @Halfak In CR, we were discussing the prediction vs. probability fields. I'm currently planning to only include the probabilities, since t... [19:40:11] (03CR) 10Joal: "> Patch Set 2:" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [19:42:03] (03CR) 10Ottomata: "Are there any potential other ores tables we might make in the future? ores.score is not bad. Also, this is pretty specifically a revisi" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [19:43:33] (03CR) 10Joal: "Ok for ores.revision_score ,I assume we'll have at some point ores.revision_features?" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [19:43:44] 10Analytics, 10Operations, 10Performance-Team, 10Traffic, 10Patch-For-Review: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Krinkle) I'm unfamiliar with the complexity needed in VCL to make this work, but if at all feasible, I think we shou... [19:48:37] 10Analytics: Create Hive UDF to compare DateTimes with partition columns - https://phabricator.wikimedia.org/T212451 (10mforns) [19:48:46] 10Analytics, 10EventBus, 10WMF-JobQueue, 10Wikimedia-production-error: EventBus error "Unable to deliver all events: (curl error: 28) Timeout was reached" - https://phabricator.wikimedia.org/T204183 (10Krinkle) 05duplicate→03Open Still seen regularly. With bursts throughout the last 12 days with about... [19:48:53] 10Analytics, 10EventBus, 10WMF-JobQueue, 10Wikimedia-production-error: EventBus error "Unable to deliver all events: (curl error: 28) Timeout was reached" - https://phabricator.wikimedia.org/T204183 (10Krinkle) p:05Triage→03High [19:49:33] 10Analytics: Create Hive UDF to compare DateTimes with partition columns - https://phabricator.wikimedia.org/T212451 (10mforns) Note that the example code considers both since and until DateTimes inclusive, we might want to consider whether until should be exclusive? [19:50:12] 10Analytics: Create Hive UDF to compare DateTimes with partition columns - https://phabricator.wikimedia.org/T212451 (10mforns) Also, seems that calling UDFs from Spark SQL is possible no? https://stackoverflow.com/questions/40369170/registering-hive-custom-udf-with-spark-spark-sql-2-0-0 [19:50:31] 10Analytics, 10Core Platform Team, 10EventBus, 10WMF-JobQueue, 10Wikimedia-production-error: EventBus error "Unable to deliver all events: (curl error: 28) Timeout was reached" - https://phabricator.wikimedia.org/T204183 (10Krinkle) [19:54:53] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current): Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Halfak) I think the "attractive nuisance" of the "prediction" field is debatable for editquality and probably not applicable to other models. It f... [19:58:24] hey fdans yt? [20:04:39] nm :) [20:10:16] joal: yes [20:10:50] Heya nuria - I was reading your code for the classifier and couldn't find the select_training_data.hql file :S [20:11:56] nuria: is it the select_testing_data.hql one? [20:12:46] Completely unrelated: the MWH job with prod-comments is successfull :) The checker confirms data is ok, and it contains comments [20:12:55] pffff - took time :) [20:13:05] joal: yes, select_testing.hql : https://github.com/nuria/study/blob/master/poc-bot-classifier/select_testing_data.hql [20:13:20] joal: sorrym, this is still very poc-ish (dates of select might differ) [20:13:21] nuria: Ok - Was not sure :) [20:13:34] nuria: no prob, I prefered to confirm :) [20:13:49] milimetric: do you have a minute for me on MWH-data-quality? [20:14:12] let's chat joal [20:15:01] joal: the https://github.com/nuria/study/blob/master/poc-bot-classifier/select_testing_negative_data_sets_human.hql [20:15:11] joal: plays the part of labeled dataset for humans [20:15:37] milimetric: So IIUC the plan is to collaborate the 3 of us beginning of next month to solve the timestamp-issue and the fake-and-real-ids issue - right? [20:16:45] oh joal sorry, I was waiting in the cave [20:17:04] Ah joining milimetric - easier talking - but chat means writing to me :) [20:17:19] (03CR) 10Awight: "> Are there any potential other ores tables we might make in the" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [20:22:07] (03CR) 10Nuria: "Please also document data in wikitech, see examples: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Pageview_hourly" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [20:25:05] (03PS3) 10Awight: Schema for ORES scores [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) [20:35:00] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current): Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) [20:40:55] joal: when you left she said "more" [20:40:58] she misses you [20:41:04] huhu :) [20:41:12] I'll say hello everytime she wants :) [20:41:16] * joal loves babies :) [20:45:31] * nuria loves babies TOO [20:49:32] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current): Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) @JAllemandou I need another clue here, this query works when I run each insert separately but when run together I get an error about 'No pa... [20:53:27] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current): Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Nuria) @awight maybe moving to IRC for this might help, i think your query is missing a ";" [20:55:38] o/ Thanks for all the help :) [20:55:53] nuria: I'm trying to follow the syntax documented here, https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingdataintoHiveTablesfromqueries [20:56:01] Hi awight - Was starting to write you on the task, but IRC could indeed be easier [20:56:24] awight: The problem you're having is related to dynamic partitioning in Hive [20:56:51] minor note, I'm running: set hive.exec.dynamic.partition.mode = nonstrict; [20:56:55] awight: Assuming you have created your table with a schema and all, you want to insert new partitions into it [20:58:16] With the above setting, that part seems to be working well, the partitions are inserted based on the value supplied in the insert..select field [20:58:22] awight: normally running in nonstrict mode should do [20:58:45] awight: indeed - With the fact you have them correctly listed last in your query [21:00:42] awight: From a syntax perspective I don't know how hive reacts with a partitioned table used on multi-queries [21:01:13] hmm okay I can go ahead and run as two queries until this is worked out, anyway. [21:01:20] awight: I suggest moving the 'where' clause to the top (adding partition predicate), but if it fails syntaxically, cant really help [21:01:42] awight: sorry to have taken that time to understand the real issue :) [21:01:58] nothing to apologize for, the support is amazing! [21:02:11] Glad to help :) [21:02:43] And also thrilled to see some ores coming to the cluster :) [21:02:49] My next step is to set up a periodic job which only imports the latest rows from the event table. Are there any examples I should look for? [21:02:53] \o/ [21:03:08] o/ halfak :) [21:04:23] Yeah I think the dump files will be popular, and hopefully the scores in hive will be useful internally. [21:04:24] awight: thinking in that direction - Hadoop dislike small files, meaning we'd rather go for longer timeframes with more data - Or we can also recombine one in a while the small data into bigger one [21:04:45] That's fine for us, this will be an offline analytical store [21:05:18] awight: We handle periodic jobs with oozie - And as you've been told already - oozie is noone good friend :) [21:05:44] hehe, oh dear. [21:06:07] awight: Given you'll have hive queries, you can look at the oozie/pageview/hourly folder in analytics-refinery repo [21:06:14] This is one of the many examples [21:06:26] the input data has timestamps and is partitioned by time, so I guess I can just grab "yesterday"'s data, but I'm not sure how we handle missed jobs. [21:06:29] okay thanks! [21:07:34] awight: Trying to fasttrack some concepts here: oozie instanciates scheduled jobs (you pick the recurrence you wnat), and then waits for data-dependencies to be available to actually run them [21:08:47] The data-dependency bit happens using datasets.xml files - For pageviews for instance, the dataset the job depends on is webrequest - dataset definition is in webrequest folder in dataset.xml file [21:08:51] Makes sense awight ? [21:09:32] And the dependency definition is done in coordinator.xml, referencing datasets defined in dataset.xml files [21:10:10] thanks so much, I was concerned seeing mark_directory_done_workflow_file but you've left me lots of breadcrumbs, I'm sure I can make some progress. [21:10:18] Running away for now :) [21:10:28] Bye awight [21:33:34] (03PS1) 10Ottomata: Use wikimedia superset fork to build_wheels. @wikimedia branch currently at 0.26.3 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/481053 [21:33:36] (03PS1) 10Ottomata: Update to build from wikimedia's superset fork [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/481054 [21:34:23] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Use wikimedia superset fork to build_wheels. @wikimedia branch currently at 0.26.3 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/481053 (owner: 10Ottomata) [21:34:29] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Update to build from wikimedia's superset fork [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/481054 (owner: 10Ottomata) [21:38:15] (03PS1) 10Ottomata: Bump to superset version 0.26.3-wikimedia1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/481056 [21:40:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10Ottomata) I just had a whack at getting a build process to work with our fork. I think it works! @elukey I haven't tested, but https://gerrit.wikimedia.org/r/#/c/analytics/superse... [22:05:34] (03CR) 10Joal: [C: 03+1] "Looks good to me :) Let's Andrew merge if he likes." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/481025 (https://phabricator.wikimedia.org/T209732) (owner: 10Awight) [22:06:15] Can't even write correctly --^ - Gone to bed ! see you tomorrow team :) [23:27:08] 10Analytics, 10Pageviews-API: enetunreach responses - https://phabricator.wikimedia.org/T212477 (10Adamwiggall)