[00:01:02] (PS4) Gergő Tisza: Track opt-out ratio [analytics/multimedia] - https://gerrit.wikimedia.org/r/143501 [03:02:50] (PS5) Gergő Tisza: Track opt-out ratio [analytics/multimedia] - https://gerrit.wikimedia.org/r/143501 [03:29:20] (PS6) Gergő Tisza: Track opt-out ratio [analytics/multimedia] - https://gerrit.wikimedia.org/r/143501 [03:31:18] (PS1) Gergő Tisza: Track opt-out ratio [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148006 [03:40:20] (PS1) Gergő Tisza: Remove generated files from source control [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148009 [06:15:52] (PS1) Gergő Tisza: Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 [06:15:58] (CR) jenkins-bot: [V: -1] Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza) [07:19:23] https://github.com/internetarchive/ia-hadoop-tools [07:35:54] (PS2) Gergő Tisza: Track opt-out ratio [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148006 [07:43:01] (PS2) Gergő Tisza: Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 [07:43:06] (CR) jenkins-bot: [V: -1] Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza) [08:05:47] (PS1) Gergő Tisza: Track loading time for MediaViewer and the file page [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148024 [08:06:02] good lord, why are you people submitting patches. [08:06:04] it's 1am, go to bed. [09:15:44] (CR) QChris: Add webrequest table schema (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/146878 (owner: Ottomata) [09:48:50] (PS3) Gergő Tisza: Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 [10:24:08] (CR) Nuria: "Made couple notes in the code. Still need to test throughly." (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/147312 (https://bugzilla.wikimedia.org/67458) (owner: Milimetric) [10:46:58] (Abandoned) QChris: Switch Hive query variables to camelCase [analytics/refinery] - https://gerrit.wikimedia.org/r/144911 (owner: QChris) [11:55:45] (CR) Gilles: [C: -1] "> uses loadEventEnd; the difference using domComplete seems to be tiny" [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza) [12:04:46] (CR) Gilles: [C: -1] "This isn't possible with the current version of limn-deploy, otherwise we would have done it already. That's because limn-deploy does the " [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148009 (owner: Gergő Tisza) [12:06:09] sad, but true gi11es [12:38:18] (CR) Gilles: [C: -1] Track opt-out ratio (1 comment) [analytics/multimedia] - https://gerrit.wikimedia.org/r/143501 (owner: Gergő Tisza) [12:52:25] milimetric: I think tgr will find the motivation for a pull request on limn-deploy :) [12:58:00] (CR) Gilles: Track loading time for MediaViewer and the file page (1 comment) [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza) [14:13:37] (CR) Nuria: Add monthly active editor (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/147312 (https://bugzilla.wikimedia.org/67458) (owner: Milimetric) [14:29:17] (PS6) Milimetric: Add monthly active editor [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/147312 (https://bugzilla.wikimedia.org/67458) [14:32:34] (PS7) Milimetric: Add rolling active editor [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/147312 (https://bugzilla.wikimedia.org/67458) [14:32:50] (CR) Milimetric: Add rolling active editor (5 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/147312 (https://bugzilla.wikimedia.org/67458) (owner: Milimetric) [14:33:40] hey tnegrin, there’s no hangout link in the calendar invite, can you add one? [14:33:44] calling you [14:33:50] on the google [14:33:56] k hang on [14:34:04] actually Skype is better [14:34:41] (PS6) Nuria: Avoid pickle max recursion while serializing chain [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/146297 (https://bugzilla.wikimedia.org/67823) (owner: Milimetric) [14:34:43] (CR) jenkins-bot: [V: -1] Avoid pickle max recursion while serializing chain [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/146297 (https://bugzilla.wikimedia.org/67823) (owner: Milimetric) [14:35:03] (PS1) QChris: Fix obsolete 'kraken' references to 'refinery' [analytics/refinery] - https://gerrit.wikimedia.org/r/148071 [14:35:05] (PS1) QChris: Switch from 'wmf' to 'wmf_raw' as hive database for raw data [analytics/refinery] - https://gerrit.wikimedia.org/r/148072 [14:35:07] (PS1) QChris: Fix hdfs data path 'external' to 'raw' [analytics/refinery] - https://gerrit.wikimedia.org/r/148073 [14:35:09] (PS1) QChris: Drop explicitly passing SerDe [analytics/refinery] - https://gerrit.wikimedia.org/r/148074 [14:35:11] (PS1) QChris: Stop zero-padding partition specifications [analytics/refinery] - https://gerrit.wikimedia.org/r/148075 [14:35:13] (PS1) QChris: Drop noop time adding from partition specification [analytics/refinery] - https://gerrit.wikimedia.org/r/148076 [14:35:15] (PS1) QChris: Stop using potentially uninitialized variables in oozie helper workflows [analytics/refinery] - https://gerrit.wikimedia.org/r/148077 [14:35:17] (PS1) QChris: Get rid of camelCase [analytics/refinery] - https://gerrit.wikimedia.org/r/148078 [14:35:19] (PS1) QChris: Add proper xml header to xml files [analytics/refinery] - https://gerrit.wikimedia.org/r/148079 [14:35:21] (PS1) QChris: Stop feeding the Oozie's site libpath twice into Oozie [analytics/refinery] - https://gerrit.wikimedia.org/r/148080 [14:49:08] (CR) Ottomata: [C: 2 V: 2] Document '.hql' filename ending for HiveQL files (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/147402 (owner: QChris) [14:49:19] (PS3) Ottomata: Remove outdated camus jar [analytics/refinery] - https://gerrit.wikimedia.org/r/147421 (owner: QChris) [14:49:24] (CR) Ottomata: [C: 2 V: 2] Remove outdated camus jar [analytics/refinery] - https://gerrit.wikimedia.org/r/147421 (owner: QChris) [14:50:35] (CR) Ottomata: [C: 2 V: 2] Ignore generated files [analytics/udplog] - https://gerrit.wikimedia.org/r/147738 (owner: QChris) [14:54:04] (CR) Ottomata: Add webrequest table schema (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/146878 (owner: Ottomata) [14:59:59] (CR) QChris: Add webrequest table schema (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/146878 (owner: Ottomata) [15:02:12] (PS2) QChris: Document CamelCase for Python class names [analytics/refinery] - https://gerrit.wikimedia.org/r/147747 [15:10:07] (PS1) QChris: Rename HiveQL files to .hql [analytics/refinery] - https://gerrit.wikimedia.org/r/148090 [15:21:47] (PS2) Ottomata: Fix obsolete 'kraken' references to 'refinery' [analytics/refinery] - https://gerrit.wikimedia.org/r/148071 (owner: QChris) [15:22:08] (CR) Ottomata: [C: 2 V: 2] "I almost submitted a patch to do this too, but didn't cause you were working on these files." [analytics/refinery] - https://gerrit.wikimedia.org/r/148071 (owner: QChris) [15:22:36] (PS2) Ottomata: Switch from 'wmf' to 'wmf_raw' as hive database for raw data [analytics/refinery] - https://gerrit.wikimedia.org/r/148072 (owner: QChris) [15:22:41] (CR) Ottomata: [C: 2 V: 2] Switch from 'wmf' to 'wmf_raw' as hive database for raw data [analytics/refinery] - https://gerrit.wikimedia.org/r/148072 (owner: QChris) [15:23:00] (PS2) Ottomata: Fix hdfs data path 'external' to 'raw' [analytics/refinery] - https://gerrit.wikimedia.org/r/148073 (owner: QChris) [15:23:05] (CR) Ottomata: [C: 2 V: 2] Fix hdfs data path 'external' to 'raw' [analytics/refinery] - https://gerrit.wikimedia.org/r/148073 (owner: QChris) [15:23:20] (PS2) Ottomata: Drop explicitly passing SerDe [analytics/refinery] - https://gerrit.wikimedia.org/r/148074 (owner: QChris) [15:23:26] (CR) Ottomata: [C: 2 V: 2] Drop explicitly passing SerDe [analytics/refinery] - https://gerrit.wikimedia.org/r/148074 (owner: QChris) [15:24:25] (PS2) Ottomata: Stop zero-padding partition specifications [analytics/refinery] - https://gerrit.wikimedia.org/r/148075 (owner: QChris) [15:27:05] (CR) Ottomata: Stop zero-padding partition specifications (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/148075 (owner: QChris) [15:27:09] yoo qchris [15:27:13] so does [15:27:19] hm, i mean [15:27:45] i'm not sure which combination of string and zero padding you are advocating [15:28:04] you think partitions should be declared as strings and added as a strings without zero padding [15:28:05] is that correct/ [15:28:07] ? [15:28:35] I'd go with ints and no zero padding. [15:28:52] They are treated as strings in some places, but [15:28:53] so, declared as ints, added as ints? [15:29:07] in general, int best models what we want to have. [15:29:23] i agree [15:29:38] By "declared" you mean "hour int" in create table? Then yes. [15:29:42] yes [15:29:48] so, what was hive doing then, it was just seeing the zero padded number and converting it to a string? [15:29:52] in the query? [15:30:11] By "added as ints" you mean when adding partitions? Hive does not care whether or not the value is quoted. it takes it as string. [15:30:16] hm [15:30:41] what's the difference in having them as ints in the create table then? [15:30:59] Currently, there is no difference in the create table. [15:31:06] Well ... hardly any. [15:31:10] so, no matter what we do, hive will treat them as strings? [15:31:23] is it possible to do where hour > 5 then? [15:31:26] When mass dropping partitions, hive fails if the partition spec is not a string. [15:31:34] will hours 10-23 be included in that? [15:31:45] oh [15:31:47] Sure. 'hour > 5' is possible when we drop zsero padding. [15:31:47] weird [15:31:52] Yes. Weird. [15:31:56] so they are not strictly string compared? [15:32:23] No. See https://wikitech.wikimedia.org/wiki/File:Hive_partition_formats.png [15:33:02] oh hm, yeah 06 and '06' would not be > 5 if it was string compared [15:33:19] Right. [15:33:31] So there is some implicit type conversion happening. [15:33:57] And I think we should choose a variant for which queries like hour > 5 work as expected. [15:34:00] so, by using strings [15:34:07] we get 05 and 5 match 05 and '05' [15:34:15] that's the only real difference in those charts [15:35:16] so, looks to me the most correct would be to use strings without zero padding [15:35:38] hm [15:35:39] maybe... [15:35:48] But zero padding kills things like hour > '5'. [15:35:56] Meh. sorry. [15:35:57] wihtout zero padding [15:36:04] Yup. [15:36:15] ja, i'm without for dropping 0 padding [15:36:32] So I am a bit torn on int vs string. [15:36:50] hmm, actually, yeah, i geuss there isn't a difference [15:37:00] expect you said that mass dropping partitions doesn't work if we defined using ints? [15:37:02] On the one hand, strings feel better, because Hive will sort them (only in ORDER BY) by strings. [15:37:21] On the other hand, using string for a number is wrong. [15:37:38] (PS3) Ottomata: Document CamelCase for Python class names [analytics/refinery] - https://gerrit.wikimedia.org/r/147747 (owner: QChris) [15:37:43] I could not get a CDH4 cluster up to test what hive did before. [15:37:46] (CR) Ottomata: [C: 2 V: 2] Document CamelCase for Python class names [analytics/refinery] - https://gerrit.wikimedia.org/r/147747 (owner: QChris) [15:38:08] But int would better model the expectations that we have on those columns. [15:38:28] well, in that we expect them to be ints [15:38:36] Yup. [15:38:40] but if we get more expected usage behavior by using strings [15:38:48] ? [15:38:49] that seems like a good thing, ja? [15:38:59] If we drop zero padding, they are alike aren't they? [15:39:03] what do you mean by sort them in order bys? [15:39:10] will ints not be sorted properly in order bys? [15:39:12] Example: [15:39:26] qchris, i agree that your table shows them alike if we drop 0 padding [15:39:29] SELECT * FROM webrequest WHERE year != 0 ORDER BY hour [15:39:43] but, you just said that order by thing, and also, what's the deal with the mass drop partitions? [15:39:59] That query will sort by hour, as if they were strings. Regardless of whether we defined them as strings or ints. [15:40:21] The mass drop thing, I'd ignore. [15:40:36] We do not mass drop partitions anyways. [15:41:02] (That would be: ALTER TABLE .. DROP PARTITION (hour > '5')) [15:41:30] the ORDER BY thing is strange and counter-intuitive for both strings and ints. [15:41:36] If we use ints, [15:41:53] people would ask us: "Why does Hive sort them wrong". And if we [15:42:13] use strings, peaple would ask us: "Why are you too stupid to prepend 0s" [15:42:26] In the latter case, we'd have to explain all the partition madness. [15:42:47] In the first case, we can just teach them to use "ORDER BY cast(hour as int)", [15:42:56] which solves the problem and orders by integer value. [15:43:08] hmm, weird [15:43:10] super werid [15:43:10] ok [15:43:14] re the drop partition thing though [15:43:19] i have wanted to use that in the past [15:43:22] but not been able to [15:43:34] it would be nice if it worked if it could [15:43:38] but hmm [15:43:48] yeah you are right, i'd rather not have to explain all that [15:43:58] It did not work reliable for strings either. But with ints, Hive gave an error message. [15:44:32] The big advantage for using ints is: [15:44:53] If Hive ever comes to grips and does treat ints right, we need not change anything. Everything just gets more intuitive. [15:44:58] true [15:45:04] ok i'm convinced, let's do ints, no zero padding [15:45:09] (But if we used strings, and they improve int handling, we do not benefit from the improvement) [15:45:14] Cool. [15:46:37] (PS7) Milimetric: Avoid pickle max recursion while serializing chain [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/146297 (https://bugzilla.wikimedia.org/67823) [15:47:09] qchris: that PS7 above has the test working on my machine [15:47:26] qchris, other thing: i'm ok with merging this [15:47:26] https://gerrit.wikimedia.org/r/#/c/103241/2 [15:47:28] the comment in the EXPERIMENT HERE section explains everything - I'll put a note in gerrit about it [15:47:28] shall I? [15:47:42] milimetric: Ok, I'll re-test it today [15:48:01] ottomata: Sure. [15:48:27] (CR) Ottomata: [C: 2 V: 2] Add test to guard against encoding mangling of filter [analytics/webstatscollector] - https://gerrit.wikimedia.org/r/103241 (https://bugzilla.wikimedia.org/58316) (owner: QChris) [15:48:32] Thanks [15:49:51] (CR) Milimetric: Avoid pickle max recursion while serializing chain (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/146297 (https://bugzilla.wikimedia.org/67823) (owner: Milimetric) [15:50:59] oh ja qchris, why year=y instead of year='yyyy'? [15:50:59] here [15:51:00] https://gerrit.wikimedia.org/r/#/c/148075/2/oozie/webrequest/partition/add/coordinator.xml [15:51:56] Otherwise we'd have 0 padding in year 123 :-) [15:52:00] It's just for consistency. [15:52:15] We do not want zero padding for month, day, hour. [15:52:24] Hence only one character there. [15:52:44] And for consistency I also switched the year to the one character variant. [15:53:12] Would you prefer to keep the yyyy? [15:53:57] i'm confused, the character variant? [15:54:02] i thought we were going with ints [15:54:06] does y render as 2014? [15:54:11] Yup [15:54:14] pok [15:54:18] yy also 2014 [15:54:23] yyy also 2014 [15:54:26] yyyy also 2014 [15:54:30] yyyyy is 02014 [15:54:36] yyyyyy is 002014 [15:54:41] yy isn't 14? [15:54:49] ok ok [15:54:54] so [15:54:56] "'year='y,'month='M,'day='d,'hour='H" [15:55:17] should the comma be in the qutoation? [15:55:25] 'year='y, [15:55:32] Whoops. Right. yy is 14. Special case. [15:55:53] if y does the right thing, I'm ok with y [15:56:09] It did for me. Let me double-check. [15:56:16] i believe ya, its cool [15:56:17] so [15:56:26] just making sure the quotes are right there [15:57:50] Mhmm. the comma before hour looks strange. Right. [15:58:17] Retrying it ... [15:59:52] "'year='y,'month='M,'day='d,'hour='H" and [15:59:57] "'year='y',month='M',day='d',hour='H" [16:00:00] do the same thing. [16:00:30] And it is less intrusive to keep the commas outside of the quotes, so I'd keep the change as is. [16:01:00] why do we use quotes at all then? [16:01:08] (Btw. this part will get removed in follow-up changes, as the workflow needs access to the individual year etc to compute the correct statistics) [16:01:35] ok [16:01:36] To allow Java to disambiguate the quoted y from the unquoted y. [16:01:36] why not do: [16:01:43] aye, just realized that [16:01:45] why not do [16:01:45] the unquoted y becomes 'year' [16:02:11] wait no, why is it less intrusive to keep commas outside of quotes? [16:02:22] your second line looks much more correct to me, even if it doesn't make a difference [16:02:26] Because the previous change had it outside too [16:02:30] oh [16:02:33] pssh! [16:02:34] haha [16:02:40] naw, fix it! [16:02:46] but, i guess [16:02:51] if this will never actually be used as is [16:02:54] then I do not care :p [16:03:10] that's what you're saying, right? [16:03:23] It works as is. And it will get killed in follow-up commits. Right. [16:03:39] This part is still from krak...^W meh. [16:05:45] ok cool [16:06:05] (PS3) Ottomata: Stop zero-padding partition specifications [analytics/refinery] - https://gerrit.wikimedia.org/r/148075 (owner: QChris) [16:06:10] (CR) Ottomata: [C: 2 V: 2] Stop zero-padding partition specifications [analytics/refinery] - https://gerrit.wikimedia.org/r/148075 (owner: QChris) [16:06:18] (PS2) Ottomata: Drop noop time adding from partition specification [analytics/refinery] - https://gerrit.wikimedia.org/r/148076 (owner: QChris) [16:06:35] (CR) Ottomata: [C: 2 V: 2] Drop noop time adding from partition specification [analytics/refinery] - https://gerrit.wikimedia.org/r/148076 (owner: QChris) [16:06:54] (PS2) Ottomata: Stop using potentially uninitialized variables in oozie helper workflows [analytics/refinery] - https://gerrit.wikimedia.org/r/148077 (owner: QChris) [16:06:56] The worst part about doing atomic commits: Getting pinged a lot :-D [16:07:00] haha [16:08:20] hmm, qchris, do we really want the job name to change for each different partition? [16:08:25] re: https://gerrit.wikimedia.org/r/#/c/148077/2 [16:08:28] i'm ok with it I think [16:08:31] just double checking... [16:08:42] good point about the variable [16:08:44] I am not sure. I'd remove thos ever changing names fully. [16:08:54] yeah [16:08:57] i'd be ok with that [16:09:05] just drop the partition_spec in the name? [16:09:11] just call it [16:09:12] hive_add_partition-${table} [16:09:12] ? [16:09:16] Yup. [16:09:20] cool with me [16:09:23] Maybe even drop ${table}. [16:09:39] It does not help much. [16:09:40] naw, i like table! that's a parameter to the workflow, right? [16:09:47] hm, it doesn't righ tnow because we are only doing webrequest [16:09:52] but it might be nice to see which is which, right? [16:10:03] Ok. Then let's keep ${table}. [16:10:13] also, btw, not sure if you are doing this in your new changes, but i think $table shoudl be fully qualified with db name, right? [16:10:18] wmf_raw.webrequest [16:10:23] rather than having two parameters [16:10:47] Ok. I can adjust for that. I prefer that too. [16:11:03] It's just that some Hive commands do not like it that way. [16:11:14] oh really? [16:11:26] if it doesn't work, i dunno then, what doesn't work that way? [16:11:27] Try finding the partition locations for a table. [16:11:43] haha, how do you even do that? [16:11:50] i htink i looked that up once, but last week was searching for that and couldn't find it [16:12:21] https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowTable/PartitionExtended [16:12:56] So something like SHOW TABLE EXTENDED IN $DATABASE LIKE $TABLE PARTITION($PARTITION_SPEC) [16:13:07] A fully qualified table is of no use there. [16:13:46] (CR) Ottomata: [C: 2] Get rid of camelCase [analytics/refinery] - https://gerrit.wikimedia.org/r/148078 (owner: QChris) [16:13:59] oh interesting [16:14:00] ok, hm [16:14:11] Edge case. For sure. [16:14:19] should we force database to be a separate parameter then? just in case we run into that in the future? [16:14:35] I am torn. [16:14:37] haha [16:14:42] :-P [16:14:53] uhmmmm [16:15:01] If a coordinator needs two tables, passing the database for both of them (as they could be different databases) is bad. [16:15:02] fq table name is much simpler... [16:15:07] yeah [16:15:13] and we will almost certaiunly have that in the future [16:15:14] Right. fq is simpler. [16:15:24] as we do etl-ish stuff into a new table [16:15:31] So let's go with fq tables. [16:15:32] ok, let's stick with fq [16:15:33] k [16:15:58] k. Will switch to that. [16:16:23] (CR) Ottomata: [C: 2] Add proper xml header to xml files [analytics/refinery] - https://gerrit.wikimedia.org/r/148079 (owner: QChris) [16:16:50] (CR) Ottomata: [C: 2] Stop feeding the Oozie's site libpath twice into Oozie [analytics/refinery] - https://gerrit.wikimedia.org/r/148080 (owner: QChris) [16:17:02] (CR) Ottomata: [C: 2] Rename HiveQL files to .hql [analytics/refinery] - https://gerrit.wikimedia.org/r/148090 (owner: QChris) [16:17:19] ok, cool, i think all are +2ed except for the workflow job name one [16:17:23] Thanks. [16:18:01] How is your schedule today (I have some more discussion around Oozie :-) ) ... would prefer to start that now, or rather later? [16:18:17] oh yeah [16:18:18] oh [16:18:19] we can do now [16:18:30] but, right after we argue about refinery- prefix for all bin scripts some more [16:18:37] our favorite subject! [16:18:42] Hahahahah. Ok. [16:18:50] So for a start: How about our handling of queues. [16:18:57] Some scripts have them. [16:19:05] wait! refinery prefix first! [16:19:14] Ok. [16:19:18] prefix it is. [16:19:25] ok, re your argument about deploy script being not refinery specific [16:19:35] that is true right now, but only beause we are deploying '*' [16:19:42] we had talked about being more specific in the future [16:19:55] if I see 'camus' in the process list, we know what it is [16:19:58] if I see 'deploy' in the process list [16:20:00] who knows what that is [16:20:03] deploy is very generic [16:20:16] If /you/ see camus, /you/ know. Do other ops too? [16:20:30] camus is a specific piece of software [16:20:32] 'deploy' is not [16:20:52] We're not running that "specific piece of software", but a wapper from refinery. [16:21:24] yes, but that's ok, in this case it doesn't matter that the wrapper is in the refinery repository, the wrapper has nothing ot do with refinery [16:21:28] we could put it in puppet if we wanted to [16:21:38] Then ... let's put it in puppet. [16:21:40] haha [16:22:05] i'm not against that, gahhhhh, except that then we have to make puppet changes to chang ethings [16:22:07] I am not kidding here :-D Let's keep refinery slim. [16:22:12] we'd probably put the .properties files in puppet then [16:22:42] That's ok. If we do not need the .properties file in hdfs, it can live in puppet. [16:22:51] ja we don't [16:22:55] huhhHHhhhh, [16:22:59] i guess soooOOOO [16:23:03] Hahahaha. [16:23:05] but [16:23:09] camus .jar is in refeinery [16:23:18] we can't/shouldn't put that in puppet [16:23:42] The puppet role that sets up camus can rely on the clone of camus being present. [16:23:55] clone of refinery? [16:23:56] s/clone of cames/clone of refinery/ [16:24:06] Yes. [16:24:08] what's the point of moving it out of refinery then? [16:24:18] just so we don't name it 'bin/refinery-camus'? :p [16:24:35] No bin/refinery-camus would be fine by me ... [16:24:43] ok ok so, i just don't htink we need a blanket rule for scripts in bin/ [16:24:44] You do not want the prefix there. [16:24:47] the rule that I would like is [16:24:52] if your script ACTS ON refinery [16:24:55] then prefix it with refinery [16:25:18] make your script descriptive about what it is for [16:25:30] the deploy script is specificly FOR deploying refinery [16:25:45] camus script is FOR running Camus [16:27:00] I like simple rules. "Every file in bin has to have 'refinery-' prefix" is a simple rule. Otherwise we constantly have to argue whether or not it acts on/for/about/within refinery. [16:27:02] sigh, i feel more strongly about camus NOT being prefixed with refinery than I do for deploy script being prefixed with refinery- :p [16:27:09] Hahaha. [16:27:32] yeah, but simple rules do not cover cases well, and bin scripts are on ok place to argue about this [16:27:40] bin scripts are for human user interfaces [16:28:08] Since you put refinery's bin in PATH ... tab completion would show you 'camus', and simple-minded-qchris would not find it, as he'd start with 'refinery-' before tab-completing [16:28:31] why would simple minded qchris be trying to prefix it with refinery for tab completion? [16:28:46] Because he'd know it's a command from refinery. [16:29:03] I guess we have 39125 rolling monthly editors as of today cc: milimetric, DarTar [16:29:09] Like everything git is "git-" [16:29:15] why would he know that? let's pretend that its not refinery's bin in path [16:29:22] but that these scripts are installed in /usr/bin [16:29:27] why would you try refinery first? [16:29:29] cc Halfak too [16:29:37] camus technically has nothing to do with refinery, except that it is in that repository [16:29:44] whereas the deploy script is for deploying refinery [16:30:03] also, git scripts are git scripts, they work with git, so it makes sense for you to expect them to start with git- [16:30:04] Wait ... we used "it's in PATH" to get the refinery prefix for "deploy". Now we using "it's not in the PATH" to avoid getting the prefix for "camus"? [16:30:09] no [16:30:15] i'm saying, its in your usual PATH [16:30:19] not a special cased PATH [16:30:29] the reason its in PATH is so you don't have to worry about where it is [16:30:40] doesn't matter if 'it is in refinery' [16:30:44] Ok. But about what you said before ... "camus technically has nothing to do with refinery" ... let's kick it out then. [16:31:12] Meh. I do not care enough. [16:31:14] haha, but why? there's no good reason to other than this naming argument, we'd have to kick out the jar too, and then we'd have to set up some random other repo for git-fat deployment [16:31:28] Ok. Let's keep it in for now. [16:31:40] Maybe a few months down the road, we'll have more cases in point. [16:31:47] hahah, hm ok [16:32:00] so we defer this argument for later then, eh?! [16:32:09] Ja. It's hard to argue if there are only 3 or 4 scripts to argue around. [16:32:14] i really like being able to point folks on the camus-users list to this camus script, some use it [16:32:32] Sounds like it needs it's own repo. [16:32:35] if it was called 'refinery-camus', then they might be more reluctant to use it [16:32:40] a repo for a script? [16:32:43] Yup. [16:32:46] ha [16:32:51] Seriously though. [16:32:54] i dunno [16:32:56] That's not that bad. [16:33:00] mayyyybe, i can get camus to upstream it? [16:33:12] That sounds ideal. [16:33:18] uhh, dunno how we'd deploy it then though... [16:33:22] i guess i could put it in git-fat somehow [16:33:23] haha [16:33:37] and then it would still be in refinery, just not directly :p [16:33:44] But that's way better. [16:33:51] Then it's the official tool to run it. [16:33:51] haha [16:34:00] bin/refinery would then be a git-fat file? [16:34:00] LIke all Ops would know it's from camus. [16:34:06] sorry [16:34:09] bin/camus [16:34:34] ok, let's get this all running in prod, doing everything we want [16:34:39] refinery set up the way we want [16:34:42] then I'll try to upstream it [16:34:42]