[04:41:09] 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10RyanSteinberg) My team discussed this today and reached consensus that comparing links with the document's hostname is preferred. T... [05:13:52] (03CR) 10Nuria: "Thanks for doing these changes!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/495141 (owner: 10Gehel) [06:41:38] joal: o/ [06:41:42] bonjour! [06:41:55] today I am afk but I just merged the firewall change for labsdb1012 [06:42:17] I checked and I can telnet from an-coord1001, an-worker1080 and stat1007 [06:42:30] so should be good to go for sqoop tests [06:42:34] sorry for the delay :( [06:42:50] (I also waited for Cloud's team review etc..) [07:32:31] (03PS1) 10Elukey: Add artifacts for Debian Buster and upgrade to 0.29rc7 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/495182 (https://phabricator.wikimedia.org/T212243) [07:41:40] (03PS2) 10Elukey: Add artifacts for Debian Buster and upgrade to 0.29rc7 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/495182 (https://phabricator.wikimedia.org/T212243) [08:01:37] Morning elukey - Enjoy the conference, I'll check and start testing :) Many many thanks [08:03:45] I confirm I can access mysql from an-coord1001 -- Will start testing :D [08:16:48] (03CR) 10Joal: "Super nice change :) We tried to differenciate fat from non-fat jars as needed (core and spark are not fat while the others are), but havi" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/495141 (owner: 10Gehel) [08:21:00] (03CR) 10Gehel: "> Patch Set 1:" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/495141 (owner: 10Gehel) [08:21:55] joal: about the patch above, let me know if you want to discuss it a bit [08:22:54] for context, the need on our side was to reuse some of that code (use refinery-hive as a maven dependency), but it is obviously problematic to have a shaded jar as a dependency [08:25:51] gehel: makes sense :) [08:26:26] gehel: my `concern` if any is about naming, and use of non-shaded jars in place where shaded is expected without failure before execution [08:26:27] I've added some rational on the CR, but maybe this should move to a phab task [08:26:38] gehel: Phab is always good :) [08:27:09] yep, obviously all the use cases need to be identified and modified if we change the naming [08:27:18] I have no idea how hard that is [08:27:55] gehel: I think it's easy, but as with easy stuff, it's probably even easier to forget on and break it :) [08:27:55] But if it is simple, I think it is better to move to what is usually considered standard instead of adding complexity in the project [08:28:01] :) [08:28:23] Ok, I'm doing an elastic prez in a bit, but I'll try to remember to open a phab task later [08:28:28] gehel: I hear the standardization rational - Will advocate in that direction with the team ;) [08:28:41] have fun! [08:28:49] so you ! [08:41:30] 10Analytics, 10Cloud-Services: Discuss labsdb visibility of rev_text_id and ar_comment - https://phabricator.wikimedia.org/T158166 (10Zache) If I remember correctly the ar_comment was visible in labsdb before moving to separate comment table. I think it would be good that the comments would be still visible... [10:09:13] joal: goooood [10:14:54] https://github.com/apache/incubator-superset/blob/master/UPDATING.md - Superset is at 0.32 [10:15:08] but their stable release is still not yet 0.29 due to licensing [10:15:09] sigh [10:46:05] :/ [11:03:32] (little pause from talks now) [11:03:33] but! [11:03:44] I am trying to use 0.29rc7 from pypi [11:03:54] that seems not working of course [11:04:02] for some dependency issues [11:14:43] (03PS1) 10Mforns: Adapt config to use new auto_find_db_shard feature [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/495199 (https://phabricator.wikimedia.org/T215289) [11:15:53] 10Analytics, 10DBA, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Marostegui) 05Open→03Resolved As per our earlier chat - this seems to be working fine after the puppet change to get the FW opened for labsdb1012 [11:17:13] (03CR) 10Elukey: "This currently leads to:" [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/495182 (https://phabricator.wikimedia.org/T212243) (owner: 10Elukey) [12:07:28] elukey, heya :] I'm having problems finding the echo_notification table in the analytics replicas. Before the migration, it was in each one of the wiki databases, i.e.: enwiki, eswiki, etc. But now I can not find it... Did it change places? [12:07:50] Is there an echo database maybe? [12:17:34] fdans: Hi! I can't recall if you're already in holidays or from next week :S [12:18:47] joal: here today! [12:18:54] fdans: Yeah :) [12:19:07] fdans: I'd need some help on deploying wikistats2 - I have questionz [12:19:55] joal: mind holding until after lunch? I’m in extranjería with Lauren, will be back in an hour [12:20:17] fdans: no problemo - ping me when you have some time ;) [12:21:10] elukey: I'm super sorry I can't recall the name of the grafan dashboard holding system metrics from hosts [12:31:58] Found it elukey, nevermind :) [13:25:55] joal: I'm back if you want to chattanooga? [13:31:10] * joal tries to find a nice nooga word to answer fdans [13:31:23] Let's batcanooga fdans! [13:31:30] omw! [14:08:50] joal: how's the testing going? [14:16:05] elukey: on big wikis (enwiki and wikidatawiki so far), using 64 mappers, time taken is roughly divided by 2.5 [14:17:12] \o/ [14:17:28] that seems promising [14:17:44] so the test increased the mappers from 10 to 64 right? [14:18:05] and the speed increase is ~2.5 times [14:18:25] do you expect more raising the mappers to say 100 or more? [14:18:25] We use 4 mappers in regular jobs (not hammer the host too much) [14:18:38] And allow for 3 jobs in parallel with 1 mapper each [14:18:56] ah yeah that can be increased now :) [14:19:12] glad that everything works fine now [14:19:17] joal: I'm sorry! I accidentally turned off my irc messages and missed your pings [14:19:21] Here the setting is 64 mappers for big wikis, and 4 mappers for small wikis with 16 of them in parallel [14:19:33] super [14:19:38] Heya milimetric - No problem :) Will deploy today [14:19:39] yes, 2.5.5 [14:20:06] npm install should not generate changes for package.json, but will probably generate changes for package-lock.json [14:20:13] elukey: I have great hope in parallelizing a lot on small wikis [14:20:25] you theoretically should push those, but ugh, that file is part of what's wrong with the world [14:20:33] :) milimetric [14:20:37] joal: next week I'll be more available to watch metrics with you while you test [14:20:47] hopefully we'll find an optimal compromise [14:21:26] I've talked with fdans about those, and I'll update the doc to: first update package.json to new version (2.5.5 here), then npm install and build (changes to packages-lock.json), push the release with those to git [14:22:06] elukey: No problem, I feel we're gonna sqoop the whole world in less than 1/2 a day :) [14:22:21] milimetric: -----^ [14:22:24] the line above [14:22:29] (03CR) 10Milimetric: [C: 03+2] Adapt config to use new auto_find_db_shard feature [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/495199 (https://phabricator.wikimedia.org/T215289) (owner: 10Mforns) [14:23:20] joal: that is the spirit :D [14:23:41] joal: you're trying to draw my attention to boring release details when you're talking about 1/2 day sqoops!! That's the exciting part [14:24:01] milimetric: sorry, I need to deploy while sqooping [14:24:19] milimetric: but yeah, sqooping now actually works ;) [14:24:53] you're talking about from the new cloud replica right? [14:24:58] I'm pretty excited about that [14:25:10] milimetric: I shouldn't tell, because I wanted to be a surprise, but I'm also actually sqooping to parquet - I want to see how much gain there in mediawiki-history job with that (expected to be significant) [14:25:20] milimetric: indeed [14:25:30] milimetric, elukey - https://grafana.wikimedia.org/d/000000274/prometheus-machine-stats?orgId=1&var-server=labsdb1012&var-datasource=eqiad%20prometheus%2Fops&from=now-6h&to=now-1m&refresh=1m [14:25:37] ou might guess when I started to sqoop [14:26:56] joal: when we initially tested, we were just looking at the speed of the sqoop itself. For that, avro was the fastest. Good call on looking at parquet, that should be a lot faster [14:27:36] milimetric: it should speed-up reading time when working with the data [14:37:25] (03PS1) 10Joal: Release 2.5.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/495235 [14:37:40] milimetric: --^ [14:37:46] if you don't mind :) [14:37:51] milimetric: I have a wonder [14:38:13] Ahhh - Actually I solved it myself :) [14:38:14] joal: looking [14:38:56] joal: hm, I'd expect that commit to also delete some files out of dist/ [14:39:04] the old bundles [14:39:11] :S [14:39:14] * joal not know [14:39:53] joal: yeah, maybe it's missing from the build step and we just do it automatically or something? [14:39:59] *manually [14:40:11] milimetric: I actually don't know how to [14:40:30] milimetric: I should have had deleted dist before building - That might have been the missing step [14:40:44] joal: is that in the deploy instructions? [14:40:48] Nope [14:40:53] ok, I'll add that, it should be [14:41:12] milimetric: I'll update the docs [14:41:25] milimetric: deleting the dist folder before building should be done right? [14:42:35] joal: I added it [14:42:53] sorry about that, I should've noticed it last time I deployed [14:42:58] milimetric: there is a bunch of changes I want to make, I'll reupdate - thanks for adding it :) [14:43:31] (03PS2) 10Joal: Release 2.5.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/495235 [14:43:45] hopefully better milimetric --^ Thanks for the good catch :) [14:45:23] (03PS3) 10Fdans: Create metrics matrix component [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 [14:45:48] (03CR) 10Milimetric: [C: 03+2] Release 2.5.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/495235 (owner: 10Joal) [14:45:58] \o/ !!! [14:46:14] * joal has deployed A WEBSITE once in his life :D [14:46:14] :) [14:46:47] oh no, now you'll never want to do any Java work again :) [14:46:56] hehehe :) [14:47:18] milimetric: while we're at it we could do a lil mergy merge on this [14:47:19] https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/479223/ [14:47:24] going to JS from Java: "wait... pain is not necessary?!" [14:47:46] fdans: you just barely missed the train though [14:47:53] or do you wanna do another deploy? [14:48:04] (03CR) 10jerkins-bot: [V: 04-1] Create metrics matrix component [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 (owner: 10Fdans) [14:48:23] milimetric: nono, it just want to capitalize on the enthusiasm about the deployment process in wikistats [14:48:24] (03Merged) 10jenkins-bot: Release 2.5.5 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/495235 (owner: 10Joal) [14:49:15] milimetric: that patch is nice and small and it's been there unloved for months [14:49:34] oh fdans I never saw it! [14:49:39] (03PS4) 10Fdans: Create metrics matrix component [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 [14:52:17] !log deployed wikistats2 2.5.5 [14:52:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:52:51] fdans: I moved your WKS2 task to done :) [14:53:11] joal: thank youuuuu [14:56:58] (03PS3) 10Milimetric: Add version number to footer [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/479223 (owner: 10Fdans) [14:57:30] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add version number to footer [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/479223 (owner: 10Fdans) [14:57:49] milimetric: thank youuuu [14:57:56] :) [15:02:15] joal: you reached a peak of ~700Mbs [15:02:29] (checked with librenms) [15:02:45] we'll see if a 10G is needed [15:03:49] elukey: probably not, but who knows how much MOAR powa the beasts holds hidden [15:04:23] milimetric, fdans : updated WKS2 deployment section - please proof-read when you want :) [15:24:54] thanks very much joal, more clear. It always helps to have new eyes on it [15:24:59] https://twitter.com/thomasfuchs/status/1103700462107164672 [15:25:50] (03PS1) 10Mforns: Adapt config to use new auto_find_db_shard feature [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/495239 (https://phabricator.wikimedia.org/T215289) [15:29:31] fdans: hahahhaha [15:31:05] hehehehe [15:31:41] 😂 [15:32:08] Gone for kids - back for standup [15:46:39] (03CR) 10Mforns: [C: 03+2] Create metrics matrix component (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 (owner: 10Fdans) [15:48:07] (03CR) 10Fdans: Create metrics matrix component (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 (owner: 10Fdans) [15:48:24] (03CR) 10Mforns: "Aarrrgghhhhhh..... I clicked the other button...." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 (owner: 10Fdans) [15:48:46] mforns: just responded. tldr, I don't mind either way [15:49:32] fdans, ok, I don;'t mind as well [15:49:54] milimetric mforns: I say let's just do it this way until people riot [15:49:55] fdans, but I think the 'All metrics...' link should not be in red no? [15:50:33] mforns: I put it in red so that it would be more visible, and it's the same red as the red in the wiki logo, buuuuut we can do other ways [15:51:04] fdans, we can add a link also to the place it was designed by Ash no? after the metrics list in each detail page [15:51:34] mmm, if it's at the top, I don't think that's needed [15:51:49] that's precious space in the sidebar, we'll need it to make filters and splits more clear [15:52:58] fdans, milimetric, ok ok let's leave it like that! [15:53:37] (03CR) 10Mforns: [C: 03+2] Create metrics matrix component [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494546 (owner: 10Fdans) [15:54:53] hm, mforns I'm gonna review that if you don't mind, I have some other thoughts [15:55:19] milimetric, oh... sorry, I though you already discussed it with fran [15:55:24] I merged that... [15:55:31] :/ [15:55:31] mforns: no, on second thought, it's ok, I'll just submit another patch [15:55:41] but yeah, there's some things that we should be a little stricter on [15:55:43] sorry for stepping over [15:55:50] no prob at all, thanks for doing the review [15:57:40] fdans: hola! will review patch later on today [15:59:25] nuria milimetric yall don't worry I have another patch for wikistats if you're feeling review hungry [15:59:47] :) [15:59:56] mforns just wants to keep all the fun to himself, WELL NOT ON MY WATCH [16:00:26] heh [16:00:28] I'm feeling wikistats hungry. I think if I'm honest with myself I'm at like 1/2 efficiency when I work on anything else, then I have a little context switch time, and then I feel like myself again on Wikistats. [16:24:55] milimetric, is it ok if I self-merge the limn-blah-data fixes? I have tested them all one by one and they are pretty stupid changes [16:28:05] mforns: yeah, sure [16:28:14] ok, milimetric, thanks :] [16:28:28] (03CR) 10Mforns: [C: 03+2] Adapt config to use new auto_find_db_shard feature [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/495199 (https://phabricator.wikimedia.org/T215289) (owner: 10Mforns) [16:28:57] (03CR) 10Mforns: [V: 03+2 C: 03+2] Adapt config to use new auto_find_db_shard feature [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/495239 (https://phabricator.wikimedia.org/T215289) (owner: 10Mforns) [16:36:37] (03PS1) 10Milimetric: Bump up the referenced node version [analytics/aqs] - 10https://gerrit.wikimedia.org/r/495255 (https://phabricator.wikimedia.org/T210706) [16:37:45] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10Milimetric) Confirmed deployment-aqs servers are behaving normally with Node 10.4 Next step: deploy to prod. I pushed a change here to bump up the node version in package.json. We coul... [16:41:49] (03CR) 10Mforns: [C: 03+1] "@Bearloga, I think it's good to go now! Thanks for the changes. But as you added a lot of 'hash' labels, I wanted to make sure that you un" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493424 (https://phabricator.wikimedia.org/T209087) (owner: 10Bearloga) [16:51:10] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update reportupdater to be able to query the new db cluster that will substitute 1002 - https://phabricator.wikimedia.org/T215289 (10mforns) Hi @chelsyx and @Amire80 When fixing these reportupdater jobs to be able to connect to the new analytics replica... [17:01:59] (03CR) 10Elukey: [C: 03+1] Bump up the referenced node version [analytics/aqs] - 10https://gerrit.wikimedia.org/r/495255 (https://phabricator.wikimedia.org/T210706) (owner: 10Milimetric) [17:02:56] a-team, standuuuupp [17:03:14] yar yar yar [17:04:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10elukey) >>! In T210706#5011348, @Milimetric wrote: > Confirmed deployment-aqs servers are behaving normally with Node 10.4 > > Next step: deploy to prod. I pushed a change here to bump... [17:05:03] ottomata: there was an interesting talk in here about moving wiki-based runbooks for $procedures in jupyter notebooks [17:05:06] really interesting [17:05:20] ops-like runbooks basically [17:05:43] 10Analytics, 10Analytics-Kanban, 10EventBus, 10MW-1.33-notes (1.33.0-wmf.21; 2019-03-12), 10Patch-For-Review: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals - https://phabricator.wikimedia.org/T215442 (10Ottomata) [17:06:07] anyway, conference almost finished, I'll log off and drive home :) [17:06:11] have a nice weekend people! [17:06:14] a-team: need to miss stanup to go to office hours with coo, sorry totally spaced out about sending an e-mail [17:06:26] no prob nuria - we have started :) [17:06:50] see you elukey - have a good weekend :) [17:08:20] 10Analytics, 10Analytics-Kanban, 10EventBus, 10MW-1.33-notes (1.33.0-wmf.21; 2019-03-12), 10Patch-For-Review: Make Refine use JSONSchemas of event data to support Map types and proper types for integers vs decimals - https://phabricator.wikimedia.org/T215442 (10Ottomata) a:05JAllemandou→03Ottomata [17:23:42] 10Analytics, 10Analytics-Kanban: Test sqooping from the new dedicated labsdb host - https://phabricator.wikimedia.org/T215550 (10JAllemandou) a:03JAllemandou [17:24:10] (03PS1) 10Joal: Update refinery sqoop to use dedicated labsdb host [analytics/refinery] - 10https://gerrit.wikimedia.org/r/495266 (https://phabricator.wikimedia.org/T215550) [17:35:58] (03CR) 10Nuria: [C: 03+1] "It is happening!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/495266 (https://phabricator.wikimedia.org/T215550) (owner: 10Joal) [17:48:03] mforns: yt? [17:48:08] yep nuria [17:48:56] i think we should add a comment here "https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging#Hashing_(and_salting)" about this issue with mikhail CR around having two ids one of which is not salted [17:49:14] mforns: your comment in "Is keeping loginSessionToken necessary after adding app_install_id? I think loginSessionToken could be used to undo app_install_id salt rotation, by linking two app_install_id hashes around the time of salt rotation." [17:49:23] nuria, aha [17:49:35] will do [17:55:31] elukey: super thanks for doing this https://gerrit.wikimedia.org/r/#/c/analytics/superset/deploy/+/495182/ [17:56:11] nuria, is that enough? https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging#Hashing_(and_salting) [17:57:12] mforns: great, i think is worth sending to analysts e-mail list so they are all aware [18:07:37] ok [18:12:46] people there were alarms for aqs/druid in #operations [18:13:01] from the graph I think it might be due to mediawiki_history being published to druid? [18:14:45] wow this is not great [18:16:07] yeah https://grafana.wikimedia.org/d/000000538/druid?refresh=1m&panelId=19&fullscreen&orgId=1 [18:16:27] now that I think about it, this has happened before.. [18:16:33] will investigate on monday [18:20:21] nuria: still a WIP patch, hope to get something working next week :D [18:32:44] (03PS3) 10Elukey: Add artifacts for Debian Buster and upgrade to 0.29rc7 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/495182 (https://phabricator.wikimedia.org/T212243) [18:48:29] (03Abandoned) 10Mforns: Add matchIf function to routes [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/494919 (https://phabricator.wikimedia.org/T187806) (owner: 10Mforns) [20:33:11] PROBLEM - Hive Server on an-coord1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hive.service.server.HiveServer2 [20:33:35] wut hm [20:35:33] PROBLEM - Hive Server on an-coord1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hive.service.server.HiveServer2 [20:36:25] PROBLEM - Hive Metastore on an-coord1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hive.metastore.HiveMetaStore [20:38:06] ottomata: I'm sorry this is me hammering too much :( [20:38:23] oh? [20:38:28] ottomata: I've calmed down - hopefully processes have not died [20:38:50] ottomata: too many parallel processors sqooping :( [20:39:20] oh ya [20:39:28] joal can you run these not on an-coord1001? [20:39:38] I can use stat1007 [20:39:41] Will do so [20:39:57] ya an-coord not good for big stuff [20:40:08] well i'm just glad this wasn't me. was running some queries but now keep getting FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient [20:40:15] obviously, it'll allow hive and all to continue working - sorrrrrry :( [20:40:24] yeah tryign to figure out, i think hiv server still down [20:40:31] kk - thanks guys! [20:40:33] Mwarf :( [20:41:09] RECOVERY - Hive Metastore on an-coord1001 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hive.metastore.HiveMetaStore [20:41:16] pffffff [20:41:27] RECOVERY - Hive Server on an-coord1001 is OK: PROCS OK: 1 process with command name java, args org.apache.hive.service.server.HiveServer2 [20:42:07] ok ssould be back [20:42:12] dunno if some jobs might have failed [20:42:34] I'll double check that ottomata - Sorry for the mess :( [20:42:43] k thanks joal np [20:42:46] thanks for checking [20:43:37] ottomata: when the machine stopped responding I realized it was too much obviously - but too late :S [20:48:32] !log Rerun webrequest-load-wf-upload-2019-3-8-19 after hive outage [20:48:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:48:46] :) [21:34:20] ottomata: andrew, for the shaded jars thing that guillaume was kind enough to change, if we change the name we need to change some references to the jar right? [21:34:28] yup [21:34:39] probably most of our jobs [21:34:49] oozie, refine, etc. [21:53:44] ottomata: ok, so i think we can do it but w/o changing names, right? [21:55:26] names nuria ? [21:55:44] ottomata: w/o changing the names of the jars [21:55:51] oh [21:55:52] yeah [21:55:55] that would be easier [21:55:59] but probably less correct sorta? [21:56:02] but def easier. [21:56:13] keep the shaded ones same names, the others -non-shaded or something [21:56:22] yes, but we cannot stop the cluster -all jobs- for a jar name change [21:56:41] right? [21:57:21] it would be a migrration, they dont' all have to be stopped [21:57:22] ya we can go with skinny jars [21:57:28] they just have to be started next time with proper version [21:57:30] proper nam* [21:57:32] name* [21:57:39] everything is versioned anyway [21:57:57] so the paths should already be explicit [21:58:10] next time we'd upgrade we just have to change paths to use new version AND new shaded jar [21:58:14] ya, seems a bit of work ... [21:58:14] new shaded name* [21:58:23] i think we don't have to do it all at once [21:58:30] just whenever restarting some job for an upgrade [21:58:33] k, sounds like you are all for it [21:58:46] i dunno, let's talk with joal maybe post standup next week :) [21:58:52] problem is that the default is changing too [21:58:57] no? [21:59:09] we have a symlink like to always link to latest, right? [21:59:22] yes, but those are only used by users, not by jobs, buuut ya! [21:59:31] we could keep the symlinks pointing at the shaded jars [21:59:34] that's no prob [21:59:45] we'd have to change the jenkins job that does that [22:14:30] ottomata: i am ok doing that if you are fine with it, let';s talk to joa [22:14:33] ottomata: i am ok doing that if you are fine with it, let';s talk to joal [22:14:39] on monday