[06:43:26] ebernhardson: o/ I noticed that the query_clicks coordinators in hue are showing again hive actions, did I miss some workflow when I submitted my patch for hive2 actions? [06:44:25] In theory no I can only see [06:44:25] oozie/popularity_score/workflow.xml: [06:44:28] oozie/query_clicks/daily/workflow.xml: [06:44:31] oozie/query_clicks/hourly/workflow.xml: [06:44:56] I don't want to be pedantic but just to make sure that your workflows are working fine with hive2 actions, because in the future we'll need to use them for kerberos :) [07:00:39] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move refinery to hive 2 actions - https://phabricator.wikimedia.org/T227257 (10elukey) Opened https://github.com/wikimedia-research/Audiences-External_automatic_translation/pull/1 [07:06:39] neilpquinn: o/ one more question - is https://hue.wikimedia.org/oozie/list_oozie_coordinator/0048633-180705103628398-oozie-oozi-C/ under your ownership now? [07:06:56] mobile_apps-uniques-by_country-daily-coord running with Chelsy's user [07:07:22] I found some oozie stuff on stat1004, not sure if the job is saved in any repo [07:07:52] i'd need two things: 1) start the job with hive2 actions 2) and with a different username [07:08:00] not urgent, when you have time :) [07:23:57] * elukey afk for a bit [07:26:07] Good morning analytics! [07:51:48] wooooooooooooowwwwww [07:51:53] joal: o/ o/ o/ o/ o/ [07:52:00] Hi elukey :) [07:52:00] \o/ [07:52:15] It's a plesure being back, granted by a bunch of emails :D [07:52:45] ahahha yes I can imagine the backlog [07:52:53] super glad that you are back! [07:53:33] :) [07:53:49] I hope it doesn't mean everything is broken :) [07:54:38] hihihi :) [07:55:41] I'm sure you've noticed, you've traded a stressed for a smiling-joal version (bug version update) [07:57:40] nono the curse didn't come back this time, all good [07:58:08] there are some interesting things to chat about ops-wise (buster, spark 2.4, kerberos, etc..) [07:58:14] but nothing broken afaics :) [07:59:07] great! New stuff [08:00:08] elukey: I can't recall, have you taken holidays already? [08:04:30] nope! Will start next monday [08:05:10] Ok - I'll try to be gentle and nice before you leave :) [08:08:03] 10Analytics: Unable to access SWAP notebooks using LDAP - https://phabricator.wikimedia.org/T230627 (10elukey) I can see the following in the jupyterhub's logs; ` elukey@notebook1003:/var/log/jupyterhub$ sudo grep -rni connie * 197759:Aug 14 15:36:08 notebook1003 jupyterhub[20825]: [W 2019-08-14 15:36:08.526 Ju... [08:12:11] elukey: question about next moves - Do we go to the Apache conf in Berlin? [08:12:27] still not sure, good question though :) [08:12:54] Let's try to decide and confirm this week before you leave maybe? [08:14:10] sure! [08:25:03] 10Analytics: Unable to access SWAP notebooks using LDAP - https://phabricator.wikimedia.org/T230627 (10elukey) @cchen a couple of questions: * have you tried to log in in other places, like for example: https://turnilo.wikimedia.org, https://yarn.wikimedia.org - do you log in fine? * can you try to ssh to noteb... [08:39:34] joal: HELLO JOSEPH [08:39:40] I missed you so much [08:40:02] fdans! Good morning :) I'm glad to read you :) [08:40:22] so glad to have you back :D [08:40:39] How are you? [08:41:08] joal: working from Stockholm for a few hours before taking the flight back to Madrid [08:41:30] we're done with wikimania adventures [08:41:57] woow - Was this year a good one fdans ? [08:42:28] joal: everything was wonderful with the exception of the food <3 [08:42:33] huhu :) [08:43:06] Could be a french statement [08:43:42] joal: I'm excited for the food next year in Bangkok :) [08:44:30] Indeed~! [08:46:34] joaaaaaaaalllll :] [08:46:50] Hi mforns :) Nice to see you there :) [08:47:40] nice to have you back joal :D [08:48:06] how was the break?? [08:48:51] mforns: Super great :) I managed to do mostly do what I wanted, and we went for 2 weeks in holidays with familly [08:49:18] niice, where? [08:50:12] mforns: Burgundy, France - There was some awesome stuff for kids (https://en.wikipedia.org/wiki/Gu%C3%A9delon_Castle for instance), and good wine for parents ;) [08:53:13] oh, interesting idea of the castle O.o [08:56:22] super fun actually to learn how to do stuff without powertools :) [09:09:01] 10Analytics: Upgrade Turnilo to its latest upstream - https://phabricator.wikimedia.org/T230709 (10elukey) p:05Triage→03Normal [09:10:13] yea, it's like learning to do wmf analytics without joseph xD [09:10:57] :D [09:16:43] 10Analytics: Upgrade Turnilo to its latest upstream - https://phabricator.wikimedia.org/T230709 (10elukey) [09:19:34] 10Analytics: VM request to swap analytics-tool1002 with its equivalent on buster - https://phabricator.wikimedia.org/T230711 (10elukey) [09:19:58] 10Analytics, 10Operations, 10vm-requests: VM request to swap analytics-tool1002 with its equivalent on buster - https://phabricator.wikimedia.org/T230711 (10elukey) [09:20:05] 10Analytics, 10Operations, 10vm-requests: VM request to swap analytics-tool1002 with its equivalent on buster - https://phabricator.wikimedia.org/T230711 (10elukey) p:05Triage→03Normal [09:20:41] 10Analytics, 10Operations, 10vm-requests: VM request to swap analytics-tool1002 with its equivalent on buster - https://phabricator.wikimedia.org/T230711 (10elukey) [09:28:01] (03PS1) 10Elukey: Upgrade turnilo to 1.17.0 [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/530813 (https://phabricator.wikimedia.org/T230709) [09:33:23] 10Analytics, 10Operations, 10vm-requests: VM request to swap analytics-tool1002 with its equivalent on buster - https://phabricator.wikimedia.org/T230711 (10elukey) ` elukey@ganeti1001:~$ sudo gnt-group list Group Nodes Instances AllocPolicy NDParams row_A 4 38 preferred ovs=False, ssh_port=22,... [09:47:57] joal: mforns btw the new glorious (kinda) wmf.mediarequests dataset is available between may and now in hive [10:13:01] I am currently building a new vm/host (an-tool1007) with Debian buster, will be the new home of turnilo [10:13:17] and also I'll deploy to it Turnilo 1.17.0 (we have 1.8.1 now) [10:34:26] 10Analytics, 10Operations, 10vm-requests, 10Patch-For-Review: VM request to swap analytics-tool1002 with its equivalent on buster - https://phabricator.wikimedia.org/T230711 (10elukey) ` elukey@cumin1001:~$ sudo cookbook sre.ganeti.makevm eqiad_A an-tool1007.eqiad.wmnet --vcpus 2 --memory 4 --disk 20 --lin... [11:07:51] 10Analytics, 10Operations, 10vm-requests, 10Patch-For-Review: VM request to swap analytics-tool1002 with its equivalent on buster - https://phabricator.wikimedia.org/T230711 (10elukey) Of course I used the wrong row, so I had to gnt-instance remove an-tool1007 and then recreate: ` elukey@cumin1001:~$ sudo... [11:36:03] this is weird [11:36:06] fdans: I think the current bug is worse for the dashboard, and there were a couple of other unrelated fixes in my patch. [11:36:06] from the new turnilo [11:36:07] Error: Cluster.url Cluster url: http://druid1001.eqiad.wmnet:8082 has invalid format. It should be http[s]://hostname[:port] [11:36:31] /o\ [11:36:50] hi joal :) welcome back [11:37:02] Hey! Hi milimetric :) Not gone yet? [11:37:08] I’m very happy you had a nice time, was reading up [11:37:18] Thank you :) [11:37:54] milimetric: I’m working on fixing it, but having all uniques as zero I think is a bigger no [11:37:57] I’m mostly gone mentally, but my body still makes the effort [11:39:30] fdans: right, oh, you’re fixing it on top of my patch? That’s totally fine then [11:40:01] milimetric: yes I only reverted the deploy [11:40:11] working on top of the patch [11:43:16] makes sense, excuse me over the next couple of days if I’m unintelligible :) [12:22:31] 10Analytics, 10Operations, 10vm-requests: VM request to swap analytics-tool1002 with its equivalent on buster - https://phabricator.wikimedia.org/T230711 (10elukey) 05Open→03Resolved [12:22:34] 10Analytics, 10Patch-For-Review: Upgrade Turnilo to its latest upstream - https://phabricator.wikimedia.org/T230709 (10elukey) [12:24:02] 10Analytics, 10Patch-For-Review: Upgrade Turnilo to its latest upstream - https://phabricator.wikimedia.org/T230709 (10elukey) Turnilo 1.17.0 is available on an-tool1007: ` ssh -L 8080:an-tool1007.eqiad.wmnet:80 an-tool1007.eqiad.wmnet ` And then localhost:8080 on the browser. If the tests are good I'll mov... [12:26:28] * elukey lunch! [12:52:33] elukey: I just created a task to transfer the ownership of that job...is that the one where the job might not be in a repo? [12:52:39] √ [12:52:42] https://phabricator.wikimedia.org/T230722 [12:59:14] 10Analytics-Kanban, 10Product-Analytics: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280 (10Milimetric) @Yair_rand, that's what we're trying to prevent, yes. The value of the data is great, and the risk will be minimized as much as possible.... [13:26:49] neilpquinn: yes exactly! [13:34:17] 10Analytics-Kanban, 10Product-Analytics: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280 (10Milimetric) I could use a collaboration on the list of countries to blacklist. The paper that Nuria mentions includes: China, Cuba, Egypt, Indonesia, I... [13:37:10] joal, I've been working on the mediawiki history dumps, and almost done... however, I still have an issue with the archive_job_output subworkflow, oozie fails when calling identify_content_file.sh. Now, when I execute that same script, with the exact params used by oozie, it works well, and I cannot find any significant error message in any log... :[ does it ring a bell to you? [13:46:20] mforns: do you have a link in hue about the failure? [13:46:47] elukey, https://hue.wikimedia.org/oozie/list_oozie_workflow/0022718-190730075836326-oozie-oozi-W/ [13:49:33] so in https://hue.wikimedia.org/oozie/list_oozie_workflow/0022730-190730075836326-oozie-oozi-W/ there is a job id [13:51:35] but if I replace job_ with application_ I get an error (not found) in the yarn uii [13:52:09] ah wait it was executed 3 days ago [13:52:26] so probably it fell out of our history mforns [13:52:43] in theory though if you re-execute and check again we should get something meaningful from the job's output [13:52:44] yes... [13:53:14] this is because we keep less state on zookeeper about past yarn application data [13:53:24] when we'll have our separate cluster it will be more [13:54:59] elukey, https://hue.wikimedia.org/oozie/list_oozie_workflow/0022726-190730075836326-oozie-oozi-W/ [13:57:51] yes that one [13:58:10] in "actions" there is job_1564562750409_69507 but I don't think we have logs for it anymore [13:58:36] elukey, I looked at those logs on friday, and there's nothing... [13:59:07] ah snap [13:59:09] but will re-run right now and get a new log [13:59:24] no no sorry then I don't have more suggestions [13:59:31] I usually check directly in the yarn logs to find clues [14:01:14] elukey, I try too, but maybe there's something I overlooked [14:11:30] mforns: I can definitely help if you want while Joseph is bootstrapping :D [14:11:43] elukey, sure! [14:11:52] it's executing, will take a bit [14:12:00] https://hue.wikimedia.org/oozie/list_oozie_workflow/0025468-190730075836326-oozie-oozi-W/?coordinator_job_id=0025467-190730075836326-oozie-oozi-C [14:15:28] Sorry for the delay mforns - I took a break after unpilling lots of emails :) [14:15:49] I'll look at the logs with you when the jobs finishes [14:16:00] joal, elukey is right, you should be able to focus on bootstrapping, not solving our issues... [14:16:08] (yet :]) [14:16:27] mforns: I'm so happy this dump is on its way :D [14:16:47] mforns: trying to help is pushing me to build up context :) [14:17:12] :) [14:28:30] 10Analytics, 10EventBus, 10Math, 10Wikimedia-Logstash, and 3 others: Restbase math server and mediawiki EventBus have conflicting log mappings for "response" - https://phabricator.wikimedia.org/T138539 (10fgiunchedi) [14:28:54] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-Logstash, 10observability: Validation error for invalid value type should include property name - https://phabricator.wikimedia.org/T116719 (10fgiunchedi) [14:28:56] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Wikimedia-Logstash, and 3 others: eventlogging syslog message not properly recognized by logstash - https://phabricator.wikimedia.org/T120874 (10fgiunchedi) [14:29:30] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Wikimedia-Logstash, and 2 others: EventBus HTTP Proxy service does not report errors to logstash - https://phabricator.wikimedia.org/T193230 (10fgiunchedi) [14:30:08] 10Analytics, 10EventBus, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Type collisions in log events causing indexing failures in ELK Elasticsearch - https://phabricator.wikimedia.org/T150106 (10fgiunchedi) [14:34:52] 10Analytics-Kanban, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Make Logstash consume from Kafka:eventlogging_EventError {Oryx} [8 pts] - https://phabricator.wikimedia.org/T113627 (10fgiunchedi) [14:35:33] 10Analytics-Engineering, 10Operations, 10Wikimedia-Logstash, 10observability: Convert Hadoop-Logstash logging to use Redis to address failures - https://phabricator.wikimedia.org/T85015 (10fgiunchedi) [14:37:01] 10Analytics-Engineering, 10Wikimedia-Logstash, 10observability: Zookeeper logging to Logstash - https://phabricator.wikimedia.org/T84908 (10fgiunchedi) [14:37:05] 10Analytics, 10Wikimedia-Logstash, 10observability: Kafka logging to Logstash - https://phabricator.wikimedia.org/T84907 (10fgiunchedi) [14:42:56] joal, elukey, the job has finished (only cawiki and rowiki) [14:43:17] ok mforns [14:44:13] https://hue.wikimedia.org/oozie/list_oozie_workflow/0025497-190730075836326-oozie-oozi-W/ [14:47:23] * joal hadn't log in stat1004 for a long time (Last login: Thu Jul 4 07:10:28 2019) [14:48:31] oh, I found sth: [14:48:32] [19/Aug/2019 14:44:00 +0000] exceptions_renderable ERROR Potential trace: [('/usr/lib/hue/apps/filebrowser/src/filebrowser/views.py', 198, 'view', 'stats = request.fs.stats(path)'), ('/usr/lib/hue/desktop/core/src/desktop/lib/fs/proxyfs.py', 118, 'stats', 'return self._get_fs(path).stats(path)'), ('/usr/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py', 291, 'stats', 'res = self._stats(path)'), ('/usr/lib/hue/desktop [14:48:33] /libs/hadoop/src/hadoop/fs/webhdfs.py', 285, '_stats', 'raise ex')] [14:48:33] [19/Aug/2019 14:44:00 +0000] exceptions_renderable ERROR Potential detail: HTTPConnectionPool(host='localhost', port=50070): Max retries exceeded with url: /webhdfs/v1/user/hue/oozie/workspaces/hue-oozie-1452553957.19/identify_content_file.sh?op=GETFILESTATUS&user.name=hue&doas=mforns (Caused by NewConnectionError(': Failed to establish a new [14:48:35] connection: [Errno 111] Connection refused',)) [14:49:48] mforns: you said that this works when you execute it? [14:50:07] yes! [14:50:08] same params [14:50:39] from what host? [14:50:47] have you tried from a worker node? [14:50:48] stat1007 [14:50:54] no... [14:51:21] no interesting log indeed: sudo -u hdfs yarn logs --applicationId application_1564562750409_77449 --appOwner mforns [14:51:39] yes, log is no bueno [14:52:13] elukey, trying to run bash script from worker node [14:52:26] elukey, any preference on what node to use? [14:52:34] anyone [14:52:39] k [14:56:22] mforns: another thing - is the bash script logging stuff to stdout? it might be useful to understand if something is done vs the script fails to start for some reason [14:56:28] 10Analytics, 10Analytics-SWAP, 10Product-Analytics: Upgrade all SWAP users to JupyterLab 1.0 - https://phabricator.wikimedia.org/T230724 (10Neil_P._Quinn_WMF) [14:56:39] aha [14:58:52] elukey, where is refinery deployed to in the nodes? [14:59:25] it is not, it is on hdfs [14:59:40] you'd need to copy the script over there [15:00:06] but logging to stdout and re-execute might help more mforns [15:00:14] if you need refinery copied it might be painful [15:00:41] elukey, but the script already prints results to stdout [15:00:52] they are collected by oozie (in theory) [15:01:11] ah yes but I meant while it runs [15:01:24] IIUC it prints only at the end right? [15:01:35] mforns: not directly related to solving the issue, but more braodly to the job: why an oozie loop? [15:01:47] standup people :) [15:01:50] Ah! [15:02:15] oh! [15:11:18] (03PS1) 10Milimetric: [WIP] draft of outputting druid geoeditor queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530878 (https://phabricator.wikimedia.org/T131280) [15:28:39] 10Analytics, 10Analytics-SWAP, 10Product-Analytics: Upgrade all SWAP users to JupyterLab 1.0 - https://phabricator.wikimedia.org/T230724 (10elukey) Hey Neil, is it something for us (as Analytics) to track or do you need any help/action from us? Not super clear what we'd need to do for JupyterLab.. [15:29:12] 10Analytics, 10Analytics-SWAP, 10Product-Analytics: Upgrade all SWAP users to JupyterLab 1.0 - https://phabricator.wikimedia.org/T230724 (10elukey) p:05Triage→03Normal [15:31:11] elukey: hmm, maybe i did something wrong when redeploying...will check [15:32:35] 10Analytics: Unable to access SWAP notebooks using LDAP - https://phabricator.wikimedia.org/T230627 (10elukey) p:05Triage→03Normal [15:33:34] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-Logstash, 10observability: Validation error for invalid value type should include property name - https://phabricator.wikimedia.org/T116719 (10elukey) p:05Triage→03Normal [15:35:35] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: API Request for unique devices for all wikipedia families is only showing data up to November 2018 - https://phabricator.wikimedia.org/T229254 (10elukey) p:05Unbreak!→03High [15:42:07] 10Analytics, 10Analytics-SWAP, 10Product-Analytics: Upgrade all SWAP users to JupyterLab 1.0 - https://phabricator.wikimedia.org/T230724 (10Neil_P._Quinn_WMF) @elukey this is meant as something for y'all to do (I mentioned it in our last hangtime). We users don't have the ability to force upgrade everyone 😁 [15:48:41] 10Analytics, 10Operations: Access to HUE for Mayakpwiki - https://phabricator.wikimedia.org/T229143 (10Neil_P._Quinn_WMF) Let me just support Maya's request here. I work primarily in JupyterLab, but I still use Hue frequently for various things: * Running quick queries or exploring the Data Lake (since Hue has... [15:54:14] 10Analytics, 10ChangeProp, 10Discovery-Search, 10EventBus, and 3 others: Better way to pause writes on elasticsearch - https://phabricator.wikimedia.org/T230730 (10mobrovac) There already is a mechanism in change propagation to back off and wait / retry later. We use this when MW is set to read-only. If th... [15:55:21] * elukey afk for a bit! [16:26:59] 10Analytics: Unable to access SWAP notebooks using LDAP - https://phabricator.wikimedia.org/T230627 (10cchen) @elukey Thanks for the updates. - I was able to log into https://turnilo.wikimedia.org but not https://yarn.wikimedia.org. -I just tried the second point you suggested, still got permission denied. [16:28:32] 10Analytics, 10Performance-Team, 10Research, 10Security-Team, 10WMF-Legal: A Large-scale Study of Wikipedia Users' Quality of Experience: data release - https://phabricator.wikimedia.org/T217318 (10Gilles) Well... the researchers ended up redacting their journal submission to reflect the fact that they c... [16:42:43] 10Analytics: Unable to access SWAP notebooks using LDAP - https://phabricator.wikimedia.org/T230627 (10elukey) In fact for Yarn I can see: ` /var/log/apache2/yarn.wikimedia.org.log:1:[Mon Aug 19 16:11:46.852042 2019] [auth_basic:error] [pid 7525] [client 10.64.16.22:6265] AH01617: user conniecc1: authentication... [16:50:13] 10Analytics, 10Analytics-SWAP, 10Product-Analytics: Upgrade all SWAP users to JupyterLab 1.0 - https://phabricator.wikimedia.org/T230724 (10elukey) Sure, I have no idea about jupyterlab so I'll have to dig a bit more into how it is installed/deployed/upgraded etc.. this is why I was asking :) [16:52:46] mforns: o/ [16:52:55] in meeting elukey [16:52:59] ah sorry [16:54:00] 10Analytics: Unable to access SWAP notebooks using LDAP - https://phabricator.wikimedia.org/T230627 (10cchen) @elukey looks like it's still not working :( [16:54:44] cchen_: o/ [16:54:54] have you tried the uppercase username in yarn? [16:55:03] from the logs I can see only the lowercase one [16:58:59] gone for diner team - back after [16:59:13] * elukey off, will read later on o/ [17:18:46] 10Analytics, 10ChangeProp, 10Discovery-Search, 10EventBus, and 3 others: Better way to pause writes on elasticsearch - https://phabricator.wikimedia.org/T230730 (10kchapman) p:05Triage→03Normal [17:21:04] /wmf/data/event/mediawiki_cirrussearch_request/datacenter=eqiad/year=2019/month=8/ [17:21:50] bah, i mean this data is missing, how to backfill? Create ticket? /wmf/data/event/mediawiki_cirrussearch_request/datacenter=eqiad/year=2019/month=8/day=13/hour=11 [17:50:01] 10Analytics: Apply hive2-server fix to command line - https://phabricator.wikimedia.org/T230741 (10mforns) [17:56:04] @elukey just tried the uppercase username, still not working 😥 [17:56:11] 10Analytics: Ensure Wikitech page about custom jupyter notebooks exists and is up to date - https://phabricator.wikimedia.org/T230742 (10mforns) [17:57:00] 10Analytics: Create a repository and user for Product Analytics Oozie jobs? - https://phabricator.wikimedia.org/T230743 (10mforns) [17:58:10] 10Analytics, 10Product-Analytics: Create a repository and user for Product Analytics Oozie jobs? - https://phabricator.wikimedia.org/T230743 (10mforns) [17:58:40] 10Analytics, 10Product-Analytics: Ensure Wikitech page about custom jupyter notebooks exists and is up to date - https://phabricator.wikimedia.org/T230742 (10mforns) [17:58:58] 10Analytics, 10Product-Analytics: Apply hive2-server fix to command line - https://phabricator.wikimedia.org/T230741 (10mforns) [18:03:21] I'm trying to set up JetBrains DataGrip as a local IDE alternative to Hue. I've made an SSH tunnel and have checked out the beeline wrapper script on stat1007 but can't figure out how to get past the password prompt when connecting. [18:03:56] joal mforns: would either of you have a moment to help with this? the product analytics team is very interested in this [18:04:07] (but no rush) [18:08:48] Hi, likewise I am facing similar issues as Mikhail mentioned above! [18:18:29] bearloga: make sure you use /usr/local/bin/beeline and not /usr/bin/beeline [18:19:01] bearloga: the one in /usr/local/ sets some default options like user, metastore uri, etc. might do the trick? [18:20:15] ebernhardson: yup, using some of the info from /usr/local/bin/beeline but there's nothing about the password [18:23:54] I've got `ssh -N stat7 -L 10000:analytics1003.eqiad.wmnet:10000` tunnel open but Maya and I are stuck at this prompt. https://usercontent.irccloud-cdn.com/file/os0oPMJx/Screen%20Shot%202019-08-19%20at%202.22.09%20PM.png [18:26:04] hm… `beeline` logs `issuing: !connect jdbc:hive2://an-coord1001.eqiad.wmnet:10000 bearloga [passwd stripped]` on stat1007 so not sure if there's a conf file that has a password or what's going on [18:29:14] hmm :S not seeing anything obvious for where that password comes from. it's rading a few conf files but nothing with passwords in them [18:36:21] oddly we don't set any jdbc parameters, i think that means this isn't using kerberos yet. not really sure :( [18:48:13] Hi folks - Back from diner [18:48:29] bearloga: tell me more about the password thing [18:57:47] joal: welcome back! :) I posted more info. this is part where I'm stuck https://usercontent.irccloud-cdn.com/file/os0oPMJx/Screen%20Shot%202019-08-19%20at%202.22.09%20PM.png [18:58:12] this is after making a tunnel with `ssh -N stat7 -L 10000:analytics1003.eqiad.wmnet:10000` [18:59:11] bearloga: hm - analytics1003.eqiad.wmnet ? [18:59:19] joal: ebernhardson tried looking into but couldn't find anything about any password either. beeline does log `issuing: !connect jdbc:hive2://an-coord1001.eqiad.wmnet:10000 bearloga [passwd stripped]` but idk [18:59:50] yeah if I try `an-coord1001.eqiad.wmnet` it's even worse: [19:02:05] oh the extra errors went away. yeah, it's the same deal with a tunnel to `an-coord1001.eqiad.wmnet`. If I don't put anything in for the password I get: `[ 08S01] Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: java.net.SocketException: Connection reset java.net.SocketException: Connection reset.` [19:02:24] hm [19:03:23] joal: you can download DataGrip with the Foundation's JetBrains license if you want to try it yourself :) [19:03:39] bearloga: just downloaded the thing [19:03:50] bearloga: Where can I find info on the license? [19:05:22] joal: I'll PM you [19:05:36] cheers bearloga [19:10:06] bearloga: from my install, I think the error might come from a version mismatch between our hive-server version and the hive driver version provided by datagrip [19:12:54] joal: ooooh that's an interesting lead [19:36:28] hey bearloga and joal, sorry was having dinner, forgot to afk [20:11:59] (03PS2) 10Milimetric: [WIP] draft of outputting druid geoeditor queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530878 (https://phabricator.wikimedia.org/T131280) [20:15:39] Ahhhh - I finally got it to work bearloga :) [20:15:56] That was a nice way back into the thing :) [20:16:16] AWESOME!!! :D dude you rock, how did you get it working??? [20:16:26] bearloga: driver version is actually VERY xsensitive [20:17:11] bearloga: I downloaded the standalone jar file from the cluster (scp stat1004.eqiad.wmnet:/usr/lib/hive/lib/hive-jdbc-1.1.0-cdh5.16.1-standalone.jar .) and used it it as main driver [20:17:30] bearloga: You need a tunnel to an-coord1001 [20:17:45] ssh -N stat1007 -L 10000:an-coord1001.eqiad.wmnet:10000 [20:18:10] bearloga: And your jdbc connection must have a defined username (no password)9 [20:18:26] Ah - And a defined schema as well (I use default) [20:23:47] joal: I added the JDBC driver from the cluster, opened the tunnel, and specified "default" as schema. When you click TEST CONNECTION does it still ask you for password? For me it asks for password and when I don't input anything in and click OK it gives me: [20:24:05] https://www.irccloud.com/pastebin/qE6RPepR/ [20:24:11] no password asked to me :( [20:24:40] bearloga: driver order? [20:25:13] joal: are you able to have a quick video chat so I can share my screen? [20:25:13] bearloga: to the cave for a minute if you wish :) https://meet.google.com/rxb-bjxn-nip [20:25:21] haha same wavelength [20:31:25] No prob [20:31:37] I'm glad the thing work bearloga :) [20:31:41] I'll be gone rfor tonight [20:31:55] joal: thank you so much again! :D [20:32:05] joal: have a good night [20:32:10] Cheers ! [20:45:57] (03PS3) 10Milimetric: [WIP] Publish monthly geoeditor numbers [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530878 (https://phabricator.wikimedia.org/T131280) [20:52:55] 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280 (10Milimetric) My latest patchset on that change above is just a draft implementing some of the thoughts so far. It implements the f...