[01:05:27] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, Patch-For-Review: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2432256 (Legoktm) >>! In T115119#2430019, @Milimetric wrote: > Ok, @Legoktm, I thought you sa... [01:42:22] PROBLEM - YARN NodeManager Node-State on analytics1032 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:44:41] RECOVERY - YARN NodeManager Node-State on analytics1032 is OK: OK: YARN NodeManager analytics1032.eqiad.wmnet:8041 Node-State: RUNNING [04:28:58] PROBLEM - Hadoop DataNode on analytics1034 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [04:31:27] RECOVERY - Hadoop DataNode on analytics1034 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [06:44:10] Hi elukey [06:54:32] o/ [06:54:42] going to the office, brb :) [06:54:48] elukey: sure ! [08:09:07] Analytics-Cluster, Operations, Packaging: libcglib3-java replaces libcglib-java in Jessie - https://phabricator.wikimedia.org/T137791#2379016 (MoritzMuehlenhoff) It's a bit strange, the source package has also changed and it appears there's two versions of that source in Debian stretch by now: https:... [08:33:57] * elukey is checking again hdfs' configs before increasing the datanode [08:34:05] heap size [08:34:07] :) [08:34:13] okey [08:36:01] joal: do you need me for aqs? [08:36:57] no need, just feedback :) [08:42:29] sure! anything in particular? [08:42:37] how is the bulk loading working ? [08:44:13] loading works great [08:44:30] ! [08:45:29] elukey: It put quite a lot of pressure on AQS, but it loaded 1 month of data in 4 hours (2h to generate the SSTables, 2 hours to stream) [08:50:19] wooooa [08:50:22] \o/ [08:53:54] http://www.cloudera.com/documentation/enterprise/latest/topics/admin_nn_memory_config.html [08:54:01] HADOOP_HEAPSIZE sets the JVM heap size for all Hadoop project servers such as HDFS, YARN, and MapReduce. HADOOP_HEAPSIZE is an integer passed to the JVM as the maximum memory (Xmx) argument. [08:54:13] HADOOP_NAMENODE_OPTS overrides the HADOOP_HEAPSIZE Xmx value for the NameNode. [08:54:27] but what about all the other daemons? [08:54:32] Yarn seems set [08:54:40] but I am not sure about other ones [08:54:41] grrr [09:10:49] Analytics, Analytics-Cluster, EventBus, Operations, Services: Better monitoring for Zookeeper - https://phabricator.wikimedia.org/T137302#2432626 (MoritzMuehlenhoff) p:Triage>Normal [09:45:23] (PS1) Joal: Update WikidataArticlePlaceholderMetrics params [analytics/refinery/source] - https://gerrit.wikimedia.org/r/297566 [09:45:31] addshore: --^ [09:46:12] *looks* [09:46:18] addshore: I'll add some comments in the oozie job based on the changes I suggest above [09:47:03] awesome! [09:59:11] (CR) Joal: [C: -1] "Still some errors, but not far :)" (11 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/296407 (owner: Addshore) [10:42:55] joal: I have tested the heap size change in labs, basically the HADOOP_HEAP_SIZE environment var will add a Xmx2048 to both data and namenodes. HADOOP_NAMENODE_OPTS will be appened to namenode's arguments adding a Xmx4096, so the JVM will pick up the last one [10:43:12] that should be what the cdh documentation says [10:43:16] does it make sense? [10:43:24] the other daemons should work accordingly [10:43:52] if this is true, I'll need to merge and then restart all the HDFS daemons to force the new change to be picked up by the JVMs [10:44:38] I am going to wait ottomata for a final chat, it is not that urgent :) [10:45:15] * elukey lunch! [10:54:49] (CR) Addshore: [C: 1] Update WikidataArticlePlaceholderMetrics params [analytics/refinery/source] - https://gerrit.wikimedia.org/r/297566 (owner: Joal) [11:25:28] (PS4) Addshore: Ooziefy Wikidata ArticlePlaceholder Spark job [analytics/refinery] - https://gerrit.wikimedia.org/r/296407 [11:25:34] (CR) Addshore: Ooziefy Wikidata ArticlePlaceholder Spark job (11 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/296407 (owner: Addshore) [11:59:27] a-team I'm AFK for a while [12:01:19] o/ [13:26:40] Back ! [13:31:36] Hey addshore, looks like your code is ready to be tested :) [13:34:44] ooooh [13:41:22] addshore: I suspect you have never tested oozie :) [13:41:30] addshore: let me know when is a good time for you [13:41:30] nope! [13:41:43] okay, just going to go and grab a quick bite to eat! [13:41:56] addshore: take yout tiem :) [13:42:18] addshore: I have meetings 5:30 to 6:30 pm CET [13:42:27] addshore: but except from that, flexible :) [13:42:48] okay! [13:52:11] joal, do you have some minutes for scala help please? [13:52:17] mforns: I do [13:52:24] mforns: To the batcave ! [13:52:30] :] [14:09:24] ottomata: aloha! puppet disable --ANALYTICS-CLUSTER && merge && selected puppet runs to be super sure [14:09:28] ? [14:09:44] hoyo! [14:09:55] --ANALYTICS-CLUSTER [14:09:56] hheh [14:10:05] ja sounds good, the puppet runs won't restart anything though [14:10:15] so, you might be able to just merge and then run puppet and restart selectvely [14:10:38] ah yes good point, always forget [14:10:42] elukey: i saw a nodemanager flap alert last night, is it related? [14:11:01] the nodemananger heap size is already set via YARN_HEAPSIZE, ja? [14:11:02] to 2048? [14:11:32] ah snap I didn't notice the Yarn failure [14:11:37] I saw HDFS onyl [14:11:40] *only [14:11:45] havn't looked, loking [14:11:52] 1032 [14:11:54] but yeah Yarn should have 2GB [14:12:01] :/ [14:18:45] elukey: not finding much, i do see java.io.IOException: No space left on device in the recent .out file [14:18:53] but, it can't really tell if that correlates with the time [14:19:07] yeah sorry I was working on another thing, checking [14:19:16] np [14:19:26] could be from a ffew days ago when jo filled up some disks [14:19:38] there are other exceptions there too [14:19:43] ah 1032 for yarn and 1034 for hdfs [14:19:44] weird [14:20:51] don't see anything unusual in .log for that time [14:21:59] Analytics, Operations: Jmxtrans failures on Kafka hosts caused metric holes in grafana - https://phabricator.wikimedia.org/T136405#2433186 (MoritzMuehlenhoff) p:Triage>Normal [14:22:02] hm ¯\_(ツ)_/¯ [14:22:03] :) [14:22:44] ahahhahah [14:24:10] joal: I'm ready when you are! [14:25:27] ottomata: not related but.. [14:25:28] org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /var/log/hadoop-yarn/apps/hdfs/logs is exceeded: limit=1048576 items=1048576 [14:25:32] :P [14:25:48] hm [14:25:49] oh [14:25:51] my [14:26:01] is that why i see those messages about log aggregation not working? [14:26:41] no idea about what you are saying sorry :( [14:26:59] 2016-07-06 01:56:10,223 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: Log aggregation is not initialized for container_e25_1467197794735_19368_01_000020, did it fail to start? [14:28:11] hahah [14:28:12] sudo -u hdfs hdfs dfs -ls /var/log/hadoop-yarn/apps/hdfs/logs | head [14:28:12] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded [14:28:44] elukey: those are just aggregated log files after jobs finish...i'm going to just delete them all to clear them out [14:28:49] i can't even hdfs dfs -ls them [14:29:06] objection? [14:29:28] nope [14:33:48] joal: give me a poke if you become free :) [14:34:06] addshore: in impromptu meeting, will ping you [14:34:13] okay! [14:34:50] Analytics, Operations: Jmxtrans failures on Kafka hosts caused metric holes in grafana - https://phabricator.wikimedia.org/T136405#2433216 (Gehel) I'm doing a release of jmxtrans right now. This come with a few fixes to the stability of the graphite and statsd writers, including moving to a different res... [14:36:06] hmm yikes [14:36:06] sudo -u hdfs hdfs dfs -rm -R /var/log/hadoop-yarn/apps/hdfs/logs/* [14:36:07] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded [14:36:27] addshore: Ready I am ! [14:36:39] awesome! [14:38:02] ottomata: heya1 [14:38:07] hi! [14:38:15] ottomata: have already deleted the logs? [14:38:33] no [14:38:35] not able to! [14:38:36] sorry addshore, another thing I want ot mess with, will be with you in a minute [14:38:40] maybe i deleted some [14:38:42] okay! [14:38:44] but it errored [14:38:56] ottomata: yeah, my point was, let's logs from before 2016 ? [14:39:05] That would be good already [14:39:06] joal: i would if i could,b ut i can't even look into the dir [14:39:14] ottomata: I can't find anything in the logs :/ [14:39:21] i might have to delete the whole logs directory and recreate it [14:39:23] ottomata: I know ! It's kind of a known issue [14:39:26] but -Dproc_nodemanager -Xmx2048m -Dhadoop.log.dir=/var/log/hadoop-yarn [14:39:32] if i try to ls or * the dir it hangs and eventually OOMs [14:39:44] elukey: aye [14:40:04] all right merging the change [14:40:06] ottomata: I have tried that before, I think bumping java memory for the client can make work though [14:40:08] that flap last night doesn't seem to be a node manager OOM [14:40:13] ah ok [14:40:34] ottomata: need to help addshore , let me know if you need me [14:40:46] ottomata: But I'd rather not loose all our logs ;) [14:41:16] addshore: Sooooo, testing oozie :) [14:41:19] :D [14:41:31] aye [14:41:34] joal: will try that first [14:41:43] addshore: you talk with elukey about thatone, he's gonna love explaining you how much crapy the thing is :D [14:41:59] addshore, elukey : just kidding ;) [14:42:18] hah! [14:42:45] So addshore, what you need is, on stat1002, the jar you will use (the one with the code I suggested, for parameters change) [14:43:24] addshore: And, the version of the refinery code you will test (like clone the refinery repo on stat1002, and git pull the correct patch) [14:43:36] 16:42 PROBLEM - Disk space on stat1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:44:34] okay, so I'll checkout that patch and build the jar first! [14:45:05] is there anybody working on stat1002 now? [14:45:19] I can't even do df -h :P [14:45:27] im on it :O [14:46:03] it seems veeeery slow [14:46:43] elukey: 80% CPU since 10 minutes about [14:47:24] totally locked up for me now :P [14:48:02] elukey: i think its me [14:48:08] i lsed via /mnt/hdfs [14:48:09] its hanging [14:48:11] trying to kill it :/ [14:48:47] kinda knew that wo uldn't work, but didn't thikn it would lock it up [14:48:49] indeed ottomata : fuse_dfs --> 1600% ! [14:48:57] ja [14:49:25] i f umounted it [14:49:27] i think better [14:49:44] ja ok [14:49:47] addshore: better now? [14:49:51] yup! [14:50:12] ottomata: seems that the process still exists [14:50:56] the process? [14:51:13] in top, I still see it sometimes [14:51:46] hm, gone for now it seems [14:51:47] the ls process? [14:51:50] hm ok [14:52:00] sorry for the noise ottomata [14:53:42] okay joal repackaged and I have the jar! :) [14:53:50] ok great [14:54:15] addshore: then, refinery code is needed [14:54:36] addshore: with the patch you want to apply [14:54:51] okay *goes to clone that and check that out* [14:55:22] addshore: on stat1002, obvisouly ;) [14:55:57] joal: thanks for increase of heapsize tip, i can ls things. going to just try to remove old ones [14:56:07] ottomata: Awesome [14:56:41] joal: I have that! [14:56:50] (also writing all of this down) ;) [14:57:01] ottomata: You currently are working that task : T139178 [14:57:01] T139178: Cleanup terabytes of logs on hdfs - https://phabricator.wikimedia.org/T139178 [14:57:18] ottomata: Xmx updated on analytics1032, restarted yarn and hdfs [14:57:20] all good [14:57:28] ottomata: shall I assign it to you, with 5 points, and move to in-progress? [14:57:35] addshore: Great :) [14:57:48] I am thinking to restart the daemons on 1002 [14:57:56] to double check that everything is good [14:58:03] any objections? joal, ottomata [14:58:09] addshore: now you need to make the oozie folder of the refinery repo available from hadoop (meaning, copy it to hdfs) [14:58:25] elukey: good for me (it's not master, correct?) [14:58:42] nope :) [14:58:45] I'll triple check [14:59:05] addshore: My way of doing it is: I have a oozie folder in /user/joal that contains my WIP on oozie [14:59:24] addshore: I regularly wipe it and recreate, but it's always useful for testing [14:59:36] addshore: familiar with HDFS commands? [14:59:41] nope! [15:00:21] addshore: ok, hdfs dfs - usually works [15:00:23] like: [15:00:25] hdfs dfs -ls [15:00:45] this should give you the content of your home folder on hdfs (should be /user/addshore) [15:00:47] oooh, yep, okay [15:01:14] addshore: notice the home folder on hdfs being /user/NAME, and not /home/NAME [15:01:26] uhhhh joal. i just deleted them all [15:01:29] :( [15:01:30] sorry [15:01:38] was scripting a thing to delete like 3/4 of them [15:01:42] since there are no dates on the dir names [15:02:04] but i just noticed that now there is only one dir in there...so somehow i just accidentally deleted them all [15:02:19] ottomata: DOne :) [15:02:48] addshore: So, does hdfs dfs -ls works? [15:02:53] yup [15:03:06] ok, I was just checking you actually have a hdfs home ;) [15:03:21] So, to copy files to you home: [15:03:38] hdfs dfs -put /path/to/oozie /user/addshore [15:03:55] or even leave the end empty (uses home by default) [15:04:24] okay, and what is the path to oozie ? [15:04:31] *reads up * [15:04:58] addshore: the path to the oozie folder in the refinery reop [15:05:40] ahh okay! [15:06:19] addshore: the oozie folder contains the files oozie will look after when trying to run your code (based on the parameter we'll give it) [15:07:13] awesome, so now i have an oozie folder! [15:07:19] :) [15:08:14] ottomata: while you're at it, I think it would a good idea to clean the /var/log/hadoop-yarn/apps/USERNAME folders based on time watchathink? [15:08:38] addshore: Now it's about launching a oozie job using the files that are in your folder on hdfs :) [15:09:28] joal: maybe a good time to make a task to have proper yarn aggregated log rotation :) [15:09:56] ottomata: We have the one thing already, maybe reuse that one? [15:10:15] ottomata: You have deleted logs for hdfs user, but not all the rest of us ;) [15:10:15] we have one thing? [15:10:20] oh, but all of them! [15:10:31] ottomata: That leaves us about 30Tb logs ;) [15:10:31] the hdfs user has wayyy more apps than the rest of us [15:10:39] about half of it [15:10:40] jaaa [15:10:52] hmm, ok ok ok ok ok [15:10:56] gimme a few [15:11:04] still making a task for proper rotation :) [15:11:11] I looked at it a few days ago, and we created : T139178 [15:11:11] T139178: Cleanup terabytes of logs on hdfs - https://phabricator.wikimedia.org/T139178 [15:11:17] ottomata: REUSE !!! [15:11:21] :D [15:11:30] addshore: Have you had a look at the oozie page we have? [15:11:38] I don't think so! [15:11:57] addshore: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie [15:12:08] Analytics, Analytics-Cluster: Rotate YARN aggregated logs in hdfs:///var/log/hadoop-yarn/apps/$username/logs - https://phabricator.wikimedia.org/T139470#2433286 (Ottomata) [15:12:13] addshore: I let you have a glance, then come to me to actually launch the job :) [15:12:27] oh wait, I have scanned through that one before! [15:12:44] addshore: I was kinda sure I gave it to you once ;) [15:14:12] addshore: Running a oozie job is as simple as oozie job -config /path/to/file.properties run [15:14:25] addshore: BUT, we always want to override some properties :) [15:15:07] addshore: And, there is a -dryrun mode that also helps catching some errors [15:16:10] addshore: Overriding a property when launching oozie is done by -D name=value [15:16:18] can the properties be copied from when we ran spark-submit [15:16:21] ? [15:16:29] ahh, okay, so not quite :D [15:16:35] :) [15:16:47] And, you won't run spark-submit here, oozie will it for you [15:17:17] addshore: The properties you want to override are the ones defined in your coordinator.properties file [15:17:50] joal: i don't really have a good way to do this, the dirs are not done by date [15:17:55] it was easy for hdfs user [15:18:06] i was just going to delete 700K dirs [15:18:21] this will take me a while to script, will have to parse output of hdfs ls and react [15:18:21] etc. [15:18:27] soooooo, not going to do right now :) [15:18:36] ottomata: restarted all the daemons on 1002, if you want to double check [15:18:46] on 1002? [15:19:09] namenode has -Xmx4096m [15:19:10] cool [15:19:38] it has also -Xmx2048, but the last one should win.. [15:19:43] I am double checking this assumption :) [15:19:48] also https://github.com/apache/kafka/pull/1497#issuecomment-230803562 [15:19:51] :) [15:20:18] ottomata: folder names are id based: if you sort them by name should be ok for time :) [15:20:37] addshore: I'll have an exmaple for you in a minute [15:21:18] joal: is the path to the properties file a hdfs path or regular path? [15:21:27] addshore: hdfs [15:21:32] oh no sorry, regular [15:21:36] :D [15:21:50] hehe, elukey i just saw that too! [15:21:51] :) [15:21:56] so, at least oozie job -config ~/refinery/oozie/wikidata/articleplaceholder_metrics/coordinator.properties run -Dgraphite_namespace=daily.test.articleplaceholder [15:21:57] addshore: please don't try it in run mode ;) [15:22:17] joal: ja, but how many to delete? [15:22:26] addshore: Correct, but this would fail [15:22:26] joal: not saying it can't be done [15:22:33] ottomata: Right [15:22:43] ottomata: Your call on timing :) [15:22:49] ottomata: I stop bothering ;) [15:23:01] easy to script if i always delete the same number, but to do it right we have to uhhhh do it right :) [15:23:05] i think it isn't urgent [15:23:23] ottomata: The reason we filed a task is because HDFS was 65% full [15:23:31] ottomata: Having 60T logs [15:23:32] oh what task? [15:23:35] maybe i missed this [15:23:41] i thought this was mostly about too many files in one dir [15:23:43] The one I pasted two times for you already :-P [15:23:46] ...>>> [15:24:00] ottomata: T139178 [15:24:00] T139178: Cleanup terabytes of logs on hdfs - https://phabricator.wikimedia.org/T139178 [15:24:12] oh weird, haha, joal, i didn't see it because it wasn't joal [15:24:17] my brain ignored stashbot [15:24:21] huhuhu [15:24:25] np ottomata :) [15:24:58] ottomata: Aggreed, too many files wasn't good, but 30Tb logs left is kindof a lot :) [15:25:07] Analytics, Analytics-Cluster: Rotate YARN aggregated logs in hdfs:///var/log/hadoop-yarn/apps/$username/logs - https://phabricator.wikimedia.org/T139470#2433362 (Ottomata) [15:25:09] Analytics-Kanban: Cleanup terabytes of logs on hdfs - https://phabricator.wikimedia.org/T139178#2433364 (Ottomata) [15:25:42] addshore: When looking in your coordinator.properties file, you define hdfs path for the coordiantor.xml and workflow.xml files (not only) [15:25:58] addshore: For the moment, oozie wouldn't be able to find them at the place you told it [15:26:29] addshore: The easiest way to test is to override oozie_directory, using the oozie folder you created on hdfs :) [15:26:45] Analytics-Kanban: Cleanup terabytes of logs on hdfs - https://phabricator.wikimedia.org/T139178#2421739 (Ottomata) This morning I (slightly accidentally) deleted all hdfs user aggregated logs. This brought us down to 30T. We should script something to properly remove old logs for each user. Something like... [15:26:47] joal: agree cool [15:27:10] also addshore : Since you are testing, overriding start_time and end_time is kinda important, to only have a couple days run [15:28:22] ottomata, joal: rolling restart hdfs on all the analytics hosts (except 1001) [15:28:29] elukey: k [15:28:41] ahhh okay! and then that will run jobs that have no run before for that period of time? [15:29:13] addshore: You will define from which point in time you start [15:29:18] addshore: https://gist.github.com/jobar/de1adb9ac53ddd3e6be23199c14003a9 [15:29:39] addshore: I have typo (two times start_date instead of end_date) [15:29:51] addshore: But I think it should do it [15:30:13] okay! [15:30:24] addshore: the refinery_directory override is a trick to force oozie to use the latest version, even in case of broken deploy (you shouldn't mater) [15:30:27] and should I add -dryrun? and does -dryrun replace -run ? [15:30:34] yessir ! [15:30:43] First, try with dryrun instead of run [15:30:46] addshore: --^ [15:30:59] cool! [15:31:08] Error: E1002 : E1002: Invalid coordinator application URI [/user/addshore/oozie/pageview_hourly/datasets.xml], path not existed : /user/addshore/oozie/pageview_hourly/datasets.xml: /user/addshore/oozie/pageview_hourly/datasets.xml [15:31:17] mforns / joal: i'll brt [15:31:21] elukey: cool proceed [15:31:27] addshore: If oozie spits you a big bunch of XML, no easy spottable errors found, [15:31:33] addshore: Ah, errors ;) [15:32:12] Analytics, Services, cassandra, Patch-For-Review: Refactor the default cassandra monitoring into a separate class - https://phabricator.wikimedia.org/T137422#2433396 (Eevans) p:Triage>Normal [15:32:39] addshore: addshore You see the bug? [15:32:44] I didn't spot it before [15:33:05] can't imediatly spot it! :/ [15:33:06] path for dataset should be: /user/addshore/oozie/pageview/hourly/datasets.xml [15:33:28] addshore: Normal, I'm kinda use to our system and folders :) [15:34:05] ahhh, okay [15:34:46] (PS5) Addshore: Ooziefy Wikidata ArticlePlaceholder Spark job [analytics/refinery] - https://gerrit.wikimedia.org/r/296407 [15:36:36] addshore: This error is kinda ok, you don't have to re-copy the files to hdfs (since a property update [15:36:47] addshore: You could actually have tested it using -D :) [15:37:22] ahhh, true! [15:37:39] :) [15:37:52] oooooh [15:37:52] Error: E1002 : E1002: Invalid coordinator application URI [/user/addshore/oozie/pageview/hourly/datasets.xml], path not existed : /user/addshore/oozie/pageview/hourly/datasets.xml: /user/addshore/oozie/pageview/hourly/datasets.xml [15:37:58] addshore: currently in meeting, will try to followup with you but I might get long to answer [15:38:01] okay! [15:38:38] addshore: actually, I'm bad, sorry, it was /user/addshore/oozie/pageview/datasets.xml [15:38:39] /pageview/datasets.xml perhaps! [15:38:44] addshore: without the hourly [15:38:48] addshore: sorry [15:39:07] (PS6) Addshore: Ooziefy Wikidata ArticlePlaceholder Spark job [analytics/refinery] - https://gerrit.wikimedia.org/r/296407 [15:39:23] joal / mforns: uh... nvm, I'm running way late in my 1/1 [15:39:37] milimetric: ok, I'll reschedule :) [15:39:41] ok [15:46:08] addshore: I have time now [15:46:14] addshore: How is the thing goign? [15:46:15] :D [15:46:24] *switches back to the correct tabs* [15:46:32] hmmm... Error: E1002 : E1002: Invalid coordinator application URI [/user/addshore/oozie/pageview/hourly/datasets.xml], path not existed : /user/addshore/oozie/pageview/hourly/datasets.xml: /user/addshore/oozie/pageview/hourly/datasets.xml [15:46:43] oh wait, I didn't update it... [15:46:44] joal: i got a non urgent q about revision_visibility_change whenvever you have a brain context switch moment :) [15:46:47] Right, easier would be to overrid [15:46:53] yup [15:47:05] ottomata: post-standup? [15:47:12] sho [15:47:46] cool ottomata [15:48:20] joal, can a checkpoint be unpersisted? [15:48:28] joal: awesome, that gives me loads of xml! [15:48:41] mforns: I don't think, you need to remove the parent folder (for instance) [15:48:49] addshore: Your good to try in run mode :) [15:48:56] addshore: Before [15:49:02] addshore: Do you know hue ? [15:49:32] addshore: And more precisely, the oozie monitoring part of hue (this one I'm pretty sure nor ;) [15:49:35] addshore: https://hue.wikimedia.org/oozie/list_oozie_coordinators/ [15:49:50] I have logged in a few times but never really done anything on it! [15:49:54] addshore: When started, yours should come up in there [15:50:20] okay! [15:51:38] job: 0009634-160630131625562-oozie-oozi-C [15:51:42] :) [15:52:20] although it has not appeared ;) [15:52:27] hue is a bit slow [15:52:32] okay! [15:53:32] addshore: another bug :) [15:53:44] addshore: Have you found you coordinator on hue? [15:53:58] addshore: if no, refresh ;) [15:54:23] I don't see it :/ [15:54:30] addshore: hm [15:54:44] addshore: https://hue.wikimedia.org/oozie/list_oozie_coordinators/ [15:54:48]