[05:51:30] <elukey>	 o/
[06:36:55] <elukey>	 so aqs looks good afaics
[06:39:30] <elukey>	 checked also cassandra metrics for aqs1004, nothing weird found so far
[06:39:42] <elukey>	 so in ~1h I'd proceed with aqs1005 too
[06:44:07] <elukey>	 it would be great to have it reimaged by the end of the week
[06:49:08] <joal>	 +1 elukey :)
[07:14:48] <elukey>	 ah joal I forgot to mention that we are going to get 10G NICs probably with the new hadoop worker nodes :)
[07:16:20] <elukey>	 (brb for a bit!)
[07:38:04] <elukey>	 back 
[07:38:09] <elukey>	 going to reimage aqs1005!
[08:00:34] <wikibugs>	 10Analytics, 10Patch-For-Review, 10User-Elukey: Upgrade AQS to Debian Stretch - https://phabricator.wikimedia.org/T196138 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['aqs1005.eqiad.wmnet'] ```  and were **ALL** successful.
[08:10:41] <elukey>	 aqs1005 reimaged, looks good
[08:41:15] <wikibugs>	 (03CR) 10Sahil505: "I think we can merge this change now and take care of the chart (bar & line) bug in the new one :-]" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505)
[08:54:58] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (2) hardware refresh for analytics1003 - https://phabricator.wikimedia.org/T198685 (10elukey)
[09:03:53] <elukey>	 joal: everything looks stable, do you mind if I reimage aqs1006 as well ?
[09:11:08] <wikibugs>	 10Analytics, 10Patch-For-Review, 10User-Elukey: Upgrade AQS to Debian Stretch - https://phabricator.wikimedia.org/T196138 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['aqs1006.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimag...
[09:11:24] <elukey>	 (doing it now)
[09:29:46] <wikibugs>	 10Analytics, 10Patch-For-Review, 10User-Elukey: Upgrade AQS to Debian Stretch - https://phabricator.wikimedia.org/T196138 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['aqs1006.eqiad.wmnet'] ```  and were **ALL** successful.
[09:32:46] <elukey>	 aqs1006 reimaged :)
[09:33:02] <elukey>	 since the process is so smooth, I think I'll be done tomorrow
[09:50:58] <wikibugs>	 (03PS1) 10Fdans: Changes map component to accept concrete numbers [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443582
[09:51:06] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Changes map component to accept concrete numbers [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443582 (owner: 10Fdans)
[09:51:44] <wikibugs>	 (03PS2) 10Fdans: Changes map component to accept concrete numbers [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443582 (https://phabricator.wikimedia.org/T188928)
[09:51:51] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Changes map component to accept concrete numbers [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443582 (https://phabricator.wikimedia.org/T188928) (owner: 10Fdans)
[09:58:24] <wikibugs>	 (03PS3) 10Fdans: Changes map component to accept concrete numbers [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443582 (https://phabricator.wikimedia.org/T188928)
[10:04:08] <wikibugs>	 (03PS4) 10Fdans: Changes map component to accept concrete numbers [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443582 (https://phabricator.wikimedia.org/T188928)
[10:07:48] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 031] "Yea, I agree. But let's merge this change only when the other task (bug) is finished. This way, we do not block deployments (in the unlike" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505)
[10:10:21] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests: eqiad: (1) new stat box to offload users from stat1005 - https://phabricator.wikimedia.org/T196345 (10faidon) The argument that switches between stat boxes are expensive in staff time, so we should make them less often doesn't resonate much with me (maybe we sh...
[10:11:22] <wikibugs>	 10Analytics: Q1 2018/19 Analytics procurement - https://phabricator.wikimedia.org/T198694 (10elukey) p:05Triage>03Normal
[10:18:29] <wikibugs>	 10Analytics: Q1 2018/19 Analytics procurement - https://phabricator.wikimedia.org/T198694 (10elukey)
[10:18:41] <elukey>	 all hw requests listed in --^
[10:18:45] <elukey>	 so we can track their status
[10:32:37] <wikibugs>	 10Analytics: Order Data Lake Hardware - https://phabricator.wikimedia.org/T198424 (10elukey) Hardware request task for the Druid Analytics cluster: https://phabricator.wikimedia.org/T166510  3 nodes with 64G of RAM and four Intel 1.6T SSD disks. We are currently setting a 2.9T raid10 lvm ext4 partition on each n...
[10:32:56] <elukey>	 joal: --^ (whenever you have time to add comments)
[10:41:56] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Partially purge MobileWikiAppiOSUserHistory eventlogging schema - https://phabricator.wikimedia.org/T195269 (10chelsyx) @mforns > I think the graph in T130432 has changed since then. I executed the provided query and on 2017-04-20 th...
[10:57:32] <wikibugs>	 10Analytics, 10Product-Analytics, 10Reading-analysis: Assess impact of ua-parser update on core metrics - https://phabricator.wikimedia.org/T193578 (10fdans) Ping @Tbayer let's mark this as resolved?
[11:24:04] <wikibugs>	 (03PS1) 10Sahil505: Updated the functioning & styling of filter & split switches [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443592 (https://phabricator.wikimedia.org/T198183)
[11:39:44] <elukey>	 reimaging aqs1007 now! 
[12:19:08] <joal>	 Hi elukey! You're unstoppable with aqs reimaging :)
[12:19:54] <elukey>	 \o/
[12:21:35] <joal>	 elukey: I'm not sure how you'd like me to comment on presto RAM thing :)
[12:21:40] <joal>	 I'm gonna look for documentation :)
[12:23:44] <elukey>	 if it makes sense or not, your thoughts etcc..
[12:27:21] <elukey>	 joal: aqs1007 repooled!
[12:27:26] <joal>	 \o/
[12:27:26] <elukey>	 only two to go
[12:29:16] <joal>	 elukey: scanning stuff on presto - Interesting to understand parquet: https://eng.uber.com/presto/
[12:35:40] <elukey>	 "we had over two thousand people running more than one hundred thousand analytic queries daily"
[12:35:54] <joal>	 elukey: samll stuff ;)
[12:38:07] <elukey>	 joal: one qs - will the data lake in labs only run presto without any hadoop/hdfs/etc.. related thing?
[12:38:18] <joal>	 elukey: I assume so yes
[12:38:31] <joal>	 elukey: My understanding is that the data will be available on every node
[12:39:22] <joal>	 elukey: The more I read, the more I think presto uses RAM for intermediate joins/aggregation, but reads original data at every query
[12:39:59] <joal>	 I'm assuming therefore that system-cache is important in our use case (we could also test the in-memory-connector)
[12:40:36] <elukey>	 there is a memory connector (https://prestodb.io/docs/current/connector/memory.html) but I think it is not really great for our use case
[12:40:50] <elukey>	 and yes, page cache will surely help!
[12:41:26] <joal>	 elukey: memory-connector seems to be oriented toward small/medium datasets size
[12:41:59] <joal>	 also elukey - Everything I read is about collocating presto and hoddop workers
[12:42:24] <joal>	 For internal use case, could be interestng?
[12:42:49] <elukey>	 it could indeed
[12:43:40] <elukey>	 the main question mark that I see now is: do we know how presto behaves/performs on local file system, and what are the trade-offs ?
[12:44:35] <joal>	 elukey: read-throuput over many nodes is obviously a huge win
[12:44:57] <elukey>	 sure but how data is layed down across nodes?
[12:45:01] <elukey>	 same data in all of them?
[12:45:07] <joal>	 yessir
[12:45:57] <elukey>	 I would set up a test presto cluster in labs first
[12:46:08] <elukey>	 play with it and figure out what we need
[12:46:17] <elukey>	 atm I feel that we are only making suppositions
[12:46:25] <elukey>	 am I wrong?
[12:46:46] <joal>	 elukey: looks like presto doesn't have a connector for local files (except some specific htt-request logs)
[12:46:57] <joal>	 Absolutely not - Let's play in labs
[12:47:03] <elukey>	 +1
[12:47:37] <elukey>	 because it we discover that hdfs is "practically" needed, then it might require us to think about the hw that we need
[12:48:57] <joal>	 elukey: indeed !!!
[12:49:20] <joal>	 elukey: thinking about the hardware is a unquestionable requirement :)
[12:50:31] <elukey>	 ack :)
[12:54:07] <elukey>	 need to go afk for ~1h for an errand, will read later on!
[12:54:10] * elukey afk for ~1h
[12:54:26] <joal>	 ack
[12:57:58] <ottomata>	 o/ :)
[13:23:18] <mforns>	 heyaaaa :]
[13:58:18] <ottomata>	 hmm, anyone know if there is a way to configure python http response to assume responses are utf-8 and decode them?
[13:58:46] <ottomata>	 i'm getting a strange error deep in some library stuff: failed with exception: <class 'TypeError'> - 'the JSON object must be str, not 'bytes''
[13:58:57] <ottomata>	 i can tell this is happening from https://github.com/toidi/hadoop-yarn-api-python-client/blob/master/yarn_api_client/base.py#L19
[13:59:11] <ottomata>	 if i modify that code and .decode('utf-8') its fine
[13:59:18] <ottomata>	 but there are also other places around that library where that happens
[14:05:04] <wikibugs>	 10Analytics, 10EventBus, 10Services (done): Create schema evolution tests for event-schemas - https://phabricator.wikimedia.org/T133419 (10Pchelolo) 05Open>03Resolved The tests have been created a long time ago. Resolving.
[14:11:13] <wikibugs>	 10Analytics, 10EventBus, 10monitoring, 10Services (done): Clean up retry-retry Kafka topics - https://phabricator.wikimedia.org/T179958 (10Pchelolo) 05Open>03Resolved This has been done both for Kafka topics and monitoring.
[14:23:32] * elukey back
[14:23:43] <elukey>	 (took a bit more than expected)
[14:24:07] <elukey>	 ottomata: o/
[14:54:55] <wikibugs>	 (03PS1) 10Mforns: Make bar-chart and line-chart resilient to breakdowns with null values [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443640 (https://phabricator.wikimedia.org/T198630)
[14:58:49] <wikibugs>	 (03CR) 10Mforns: [C: 04-1] "Still WIP, we have to fix some bar-width problems originated by the same undefined-values concept." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443640 (https://phabricator.wikimedia.org/T198630) (owner: 10Mforns)
[15:01:21] <milimetric>	 having trouble joining the batcave
[16:07:27] <addshore>	 Hi a team! I remember talking to someone (possibly milimetric) about looking at the number of clicks on links on wikipages? was I dreaming? is there an easy to query data set for that somewhere?
[16:08:20] <milimetric>	 addshore: you can use pagelinks in combination with webrequest and the referer field to get answers but there's nothing pre-generated
[16:09:05] <addshore>	 ack
[16:10:18] <milimetric>	 addshore: also baha and leila just launched a new schema that I think tracks clicks on reference links, but that sounds different from what you need
[16:10:33] <milimetric>	 addshore: just in case you don't know, wmf_raw.mediawiki_pagelinks is sqooped monthly
[16:10:51] <addshore>	 thanks!
[16:12:01] <joal>	 addshore: might you be referring clickstream?
[16:12:20] <addshore>	 joal: perhaps, I couldn't find much on wikitech about clickstream though
[16:12:26] <joal>	 addshore: https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream
[16:12:47] <addshore>	 aaah, because it is not on wikitech!
[16:13:13] <joal>	 addshore: https://dumps.wikimedia.org/other/clickstream/readme.html
[16:13:57] <addshore>	 I also see a clickstream db in hadoop
[16:15:38] <joal>	 addshore: this database is the old one Ellery used
[16:15:59] <milimetric>	 I always forget about this dataset, thanks joal 
[16:17:05] <joal>	 addshore: newly computed data is accessible on HDFS here: /wmf/data/archive/clickstream
[16:17:25] <joal>	 np milimetric :)
[16:18:25] <addshore>	 but not directly availible as a table / db in hive? :)
[16:18:57] <joal>	 nope addshore 
[16:19:09] <joal>	 We could (should?) have done that
[16:22:33] * addshore still doesn't know how to query other data in HDFS ;)
[16:28:57] <ottomata>	 addshore get into some spark!
[16:33:54] * elukey off!
[16:34:07] <joal>	 addshore: I second ottomata here :)
[16:43:14] <wikibugs>	 (03CR) 10Sahil505: [C: 04-1] "WIP" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/437387 (https://phabricator.wikimedia.org/T190915) (owner: 10Sahil505)
[17:06:46] <addshore>	 ottomata: joal I'll have to ;)
[17:44:33] <joal>	 addshore: I have tricks for you if you wish )
[18:19:24] <mforns>	 joal, yt? :] have you done spark-scala profiling before?
[18:19:47] <joal>	 Hi mforns 
[18:20:01] <joal>	 Not very deeply no
[18:20:40] <mforns>	 joal, I'm looking into this:  https://www.linkedin.com/pulse/profiling-spark-applications-one-click-michael-spector/
[18:20:45] <mforns>	 https://raw.githubusercontent.com/spektom/spark-flamegraph/master/spark-submit-flamegraph
[18:21:52] <joal>	 mforns: I know we could also go for https://github.com/criteo/babar
[18:22:01] <mforns>	 lookin
[18:22:26] <mforns>	 that looks better
[18:23:00] <joal>	 mforns: More "integrated" I think
[18:23:05] <mforns>	 joal, suddenly the EL Sanitization is a LOT slower
[18:23:09] <joal>	 mforns: But more work to put into place
[18:23:18] <joal>	 hm mforns 
[18:23:23] <joal>	 batcave for a moment?
[18:23:27] <mforns>	 and I think it's since the logs have the geocoded_data field populated
[18:23:28] <mforns>	 sure
[18:28:59] <wikibugs>	 10Analytics, 10Product-Analytics: Upgrade SWAP's JupyterLab from - https://phabricator.wikimedia.org/T198738 (10Ottomata) Since the dependencies are installed in each user's personal virtualenv, you can actually do this yourself in the Jupyter CLI.  Launch a terminal in Jupyter and then do  ``` pip install --u...
[18:32:16] <wikibugs>	 (03PS1) 10Ottomata: Updating wheels with Apache toree 0.2.0 rc5, and jupyterlab 0.32.1 [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/443669 (https://phabricator.wikimedia.org/T198738)
[18:33:39] <wikibugs>	 (03CR) 10Ottomata: [V: 032 C: 032] Updating wheels with Apache toree 0.2.0 rc5, and jupyterlab 0.32.1 [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/443669 (https://phabricator.wikimedia.org/T198738) (owner: 10Ottomata)
[18:46:15] <wikibugs>	 (03PS1) 10Ottomata: Use locally committed toree .tar.gz to build wheel [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/443672
[18:47:06] <wikibugs>	 (03CR) 10Ottomata: [V: 032 C: 032] Use locally committed toree .tar.gz to build wheel [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/443672 (owner: 10Ottomata)
[19:41:10] <wikibugs>	 10Analytics, 10Product-Analytics, 10Patch-For-Review: Upgrade SWAP's JupyterLab from beta 1 to beta 2 - https://phabricator.wikimedia.org/T198738 (10Neil_P._Quinn_WMF) >>! In T198738#4395417, @Ottomata wrote: > Since the dependencies are installed in each user's personal virtualenv, you can actually do this...
[19:44:00] <ottomata>	 joal:  yt?
[19:44:25] <joal>	 Hi ottomata 
[19:44:28] <ottomata>	 ok great
[19:44:30] <joal>	 Yes I'm here
[19:44:33] <ottomata>	 want to try some toree stuff
[19:44:39] <ottomata>	 can you open tunnel to notebook1004?
[19:44:45] <joal>	 I'm with mforns trying to get what's wrong with sanitiation
[19:44:47] <ottomata>	 gotta run some commands and see if some kernels wil work
[19:44:50] <ottomata>	 oh ok
[19:44:52] <ottomata>	 no hurry!
[19:45:01] <joal>	 will ping back
[19:47:08] <ottomata>	 OH!  you've never logged into notebook1004 before great
[19:47:11] <ottomata>	 this is an even better test
[20:16:53] <joal>	 ottomata: wating for some job o work - Can I help?
[20:16:59] <ottomata>	 YESSSSS lets try
[20:17:07] <ottomata>	 ssh tunnel to notebook1004
[20:17:21] <ottomata>	 ssh -N notebook1004.eqiad.wmnet -L 8000:127.0.0.1:8000
[20:17:35] <ottomata>	 and let's see what kernels you have
[20:17:38] <ottomata>	 and if they work...
[20:18:19] <ottomata>	 hmm
[20:18:20] <joal>	 login in
[20:18:23] <ottomata>	 OOOO
[20:18:27] <ottomata>	 ok something won't work...
[20:19:32] <ottomata>	 ok hold on, we try again in a bit...i could just make it work, but this login part should work
[20:19:37] <ottomata>	 so lemme see if i can fix that
[20:20:33] <joal>	 np ottomata 
[20:25:50] <ottomata>	 ok joal try again plz
[20:26:32] <ottomata>	 hmm
[20:26:51] <ottomata>	 joal what do you see?
[20:26:57] <ottomata>	 if there is a logout option, or a stop my sever
[20:26:58] <ottomata>	 can you do that?
[20:27:17] <joal>	 I started server --> error 500 :(
[20:27:22] <ottomata>	 right
[20:27:28] <ottomata>	 log out if you can
[20:27:31] <joal>	 done
[20:27:36] <joal>	 login back ?
[20:27:36] <ottomata>	 ok, try logging back in
[20:27:42] <ottomata>	 ya
[20:27:53] <ottomata>	 nope
[20:27:59] <ottomata>	 ok one more
[20:28:00] <ottomata>	 can you log out?
[20:28:05] <ottomata>	 again?
[20:28:16] <ottomata>	 joal: ?
[20:28:28] <joal>	 sure
[20:28:44] <ottomata>	 done?
[20:28:45] <joal>	 done
[20:29:36] <ottomata>	 ok, try logging in again joal
[20:30:28] <ottomata>	 looking betterrrrr
[20:30:39] <ottomata>	 do you have any very enticing looking spark kernels?
[20:30:48] <joal>	 Wooow
[20:30:53] <joal>	 Let me see
[20:30:55] <ottomata>	 try em!
[20:31:05] <joal>	 any?
[20:31:27] <ottomata>	 sure
[20:31:31] <ottomata>	 the yarn ones in particular though
[20:31:33] <ottomata>	 the locals are easy
[20:31:47] <joal>	 ok - Tried a local one first :)
[20:34:19] <joal>	 ottomata: spark-local means, I run spark, but not on the cluster
[20:34:23] <joal>	 I got it now
[20:34:28] <joal>	 TRying spark yarn
[20:34:41] <joal>	 I thought spark-yarn was having the kernel run on yarn :)
[20:34:58] <ottomata>	 ah no
[20:35:06] <ottomata>	 stopped working on that, but learned a lot about how that works if we wanted to
[20:35:07] <ottomata>	 its just toree
[20:37:13] <joal>	 ottomata: Scala works <3
[20:37:20] <ottomata>	 woohoo
[20:37:23] <ottomata>	 with hive yes?
[20:37:29] <joal>	 not tested yet
[20:41:25] <ottomata>	 hmmm seeing some errors in your logs...
[20:41:32] <ottomata>	 (Too many open files)
[20:42:02] <joal>	 up
[20:42:11] <joal>	 this time it succeeded
[20:42:30] <joal>	 ottomata: However errors are not nicely reported for scala
[20:42:37] <joal>	 No big deal, but not really helpful
[20:42:54] <ottomata>	 yeah i think logs are going to be abig problem with this
[20:43:33] <ottomata>	 hm, i should make a quick link or something for folks to tail their notebook logs
[20:43:52] <joal>	 ottomata: actually this time it worked
[20:44:03] <ottomata>	 btw, if you run journalctl -f -u jupyter-joal-singleuser
[20:44:05] <ottomata>	 you can see them
[20:44:36] <ottomata>	 joal hm ok
[20:44:42] <ottomata>	 was it just a big hive query that faield?
[20:45:20] <joal>	 yessir
[20:45:28] <joal>	 in local (my bad :(
[20:45:46] <ottomata>	 ohh 
[20:45:49] <ottomata>	 interesting
[20:45:51] <ottomata>	 ok then that makes mroe sense
[20:45:56] <ottomata>	 joal anotheir thing we should try
[20:45:56] <ottomata>	 https://github.com/Brunel-Visualization/Brunel/wiki
[20:46:00] <ottomata>	 i don't fully understand how to try it out
[20:46:08] <joal>	 I've seen that yes
[20:46:10] <ottomata>	 i have installed the %%brunel magic tho
[20:46:13] <ottomata>	 before
[20:46:15] <ottomata>	 that was not hard
[20:46:18] <joal>	 ok
[20:46:28] <joal>	 Will have as look :)
[20:47:22] <ottomata>	 ok cool!  do play around with it.  if we are happy i'll install for all users and then we can announce
[20:47:42] <joal>	 that's super great :)
[20:47:47] <joal>	 Thanks a mil ottomata for that :)
[20:47:54] <ottomata>	 yaaa its very cool
[20:47:55] <ottomata>	 !
[20:48:04] <ottomata>	 with scala there i'll probably start using notebook way more
[20:49:54] <wikibugs>	 (03PS1) 10Ottomata: Install toree kernels for all users [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/443736 (https://phabricator.wikimedia.org/T190443)
[20:50:07] <joal>	 ottomata: If brunnel (or vegas) does a good job at graphs, I'll do too - Otherwise, I'll still with python I think (sparkDataframe.toPandas() is just magic)
[20:50:19] <wikibugs>	 (03CR) 10Ottomata: [V: 032 C: 032] Install toree kernels for all users [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/443736 (https://phabricator.wikimedia.org/T190443) (owner: 10Ottomata)
[20:50:34] <joal>	 ottomata: tested spark SQL - works like a charm :)
[20:50:54] <ottomata>	 yeehaw
[20:53:04] <joal>	 ottomata: Do I need the add jar for brunel, or is it done for me?
[20:54:10] <ottomata>	 you need to add
[20:54:14] <joal>	 Ah ok :)
[20:54:26] <ottomata>	 i think it will work from the url
[20:54:31] <joal>	 ok trying that
[20:54:33] <ottomata>	 if not, you can dl manually and copy to your home and do file:: url
[21:00:33] <joal>	 mforns: found --conf spark.ui.retainedJobs as well
[21:00:40] <joal>	 we should that to 100 or something
[21:01:08] <mforns>	 joal, should I add --conf spark.ui.retainedJobs=100 ?
[21:01:19] <mforns>	 k
[21:01:28] <joal>	 mforns: If you're launching new big jobs, would be interesting yes
[21:01:34] <mforns>	 done
[21:01:50] <mforns>	 the march job repair job finished with success
[21:01:55] <mforns>	 weekly is much better
[21:02:00] <joal>	 mforns: Right :)
[21:02:04] <joal>	 cool
[21:03:44] <joal>	 ottomata: can't manage to have the jar working :( 
[21:04:20] <joal>	 Ah, this time it does
[21:05:14] <ottomata>	 glad i could help!
[21:05:38] <joal>	 mwarf - seems the notebook doesn't like something
[21:05:52] <joal>	 The brunel jar magic never ends
[21:06:51] <joal>	 and I don't have rights to read logs ottomata 
[21:08:48] <ottomata>	 oh
[21:08:51] <ottomata>	 it is probably not dling
[21:08:56] <ottomata>	 joal the AddJar you mean?
[21:09:01] <joal>	 ottomata: downloaded it and managed to load
[21:09:04] <joal>	 yessir
[21:09:05] <ottomata>	 oh
[21:09:16] <joal>	 I think the restart of a toree kernel is heavily slow
[21:09:34] <ottomata>	 well, its gotta load spark session iguess
[21:09:41] <ottomata>	 so it dled
[21:09:54] <joal>	 nope, downloaded the thing manually
[21:09:59] <ottomata>	 that's as far as I got with it too, i didn't try more cause their docs weren't very clear to me how to use
[21:10:00] <ottomata>	 ah ok
[21:10:08] <ottomata>	 Jul 03 21:10:01 notebook1004 bash[1174]: Caused by: org.brunel.model.VisException: Could not find the x field 'projet' while building the visualization: element[1x0]
[21:10:15] <ottomata>	 joal, you don't have the ability to see those logs?
[21:10:22] <joal>	 nope
[21:10:24] <ottomata>	 hm
[21:10:26] <ottomata>	 that's weird
[21:10:33] <joal>	 I see them in the UI, but not with the command
[21:10:57] <ottomata>	 hm ok, gonna make a task abou tthat
[21:11:01] <ottomata>	 i think thats gonna be important
[21:12:36] <ottomata>	 so joal
[21:12:37] <ottomata>	 you can't see things like
[21:12:38] <ottomata>	 Jul 03 21:11:45 notebook1004 bash[1174]: org.brunel.model.VisException: Unknown action 'lines(?)' while parsing action text: data('df') lines x(project) y(c)
[21:12:38] <ottomata>	 ?
[21:13:15] <joal>	 I acn in the UI
[21:13:20] <ottomata>	 oh
[21:13:22] <joal>	 But I have no other access
[21:13:22] <ottomata>	 they show up there?
[21:13:25] <joal>	 They do !
[21:13:27] <ottomata>	 oh ok
[21:13:29] <ottomata>	 that's good at least
[21:13:39] <joal>	 yeah yeah - They show up in UI
[21:13:59] <joal>	 There have been 1 time when they have not, but since, everyhing is fine
[21:19:35] <joal>	 no change with brunel so far
[21:22:37] <nuria_>	 mforns: did we moved the whitelist to refinery source then?
[21:22:45] <nuria_>	 mforns: or is that code still in teh works
[21:22:50] <mforns>	 nuria_, not refinery source, but refinery
[21:22:55] <nuria_>	 mforns: ahahah
[21:23:17] <mforns>	 yes, it's under /static_data/eventlogging/whitelist.yaml
[21:24:19] <nuria_>	 mforns: right! is the mysql python running off teh other list or not running now?
[21:24:40] <mforns>	 it is running off the other whitelist that is ensured by puppet
[21:25:03] <mforns>	 we need to deploy refinery in the eventlogging hosts
[21:25:14] <mforns>	 I thought I had created that task, but I'm not sure
[21:25:59] <mforns>	 no, can not find it in phab
[21:26:05] <mforns>	 will create one now
[21:26:26] <joal>	 ottomata: looks like our proxy makes me fail to download stuff from inside the workbook :(
[21:26:37] <joal>	 ottomata: any idea on how to solve?
[21:27:25] <wikibugs>	 10Analytics: Deploy refinery to eventlogging hosts - https://phabricator.wikimedia.org/T198766 (10mforns)
[21:27:33] <mforns>	 nuria_, ^
[21:28:40] <mforns>	 we can also include the puppet change that whill call the purging script for mysql with the new whitelist path
[21:32:01] <mforns>	 updated the task description
[21:41:42] <nuria_>	 mforns: thank you
[21:42:50] <nuria_>	 mforns: and you talked with ottomata and he is ok with deploying refinery to EL hosts...
[21:43:09] <nuria_>	 mforns: also we run whitelist code in beta labs so we would need to update that too
[21:43:14] <mforns>	 nuria_, yes we talked about this, right otto?
[21:43:38] <mforns>	 I see
[22:05:11] <wikibugs>	 10Analytics, 10Cassandra, 10RESTBase-Cassandra, 10monitoring, and 2 others: Upgrade prometheus-jmx-exporter on all services using it - https://phabricator.wikimedia.org/T192948 (10Eevans) AQS has been partially upgraded:  ```name="prometheus-jmx-exporter" aqs1005.eqiad.wmnet:   Installed: 1:0.3.0-1 aqs1008...
[22:06:38] <wikibugs>	 10Analytics, 10Cassandra, 10RESTBase-Cassandra, 10monitoring, and 2 others: Upgrade prometheus-jmx-exporter on all services using it - https://phabricator.wikimedia.org/T192948 (10Eevans)
[22:06:58] <wikibugs>	 10Analytics, 10Cassandra, 10monitoring, 10Puppet, 10Services (watching): Upgrade prometheus-jmx-exporter on all services using it - https://phabricator.wikimedia.org/T192948 (10Eevans)