[15:06:07] * jeh waves to milimetric [15:06:28] hi! [15:06:49] ottomata: o/ [15:06:58] hellooo [15:07:00] if you are around, do you mind to test hue?? [15:07:00] jeh: it works! what'd you do?! [15:07:02] 10Analytics, 10Patch-For-Review: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10elukey) @MMiller_WMF @Ottomata can you try now to use hue and see if the issue is still there? [15:07:08] and thank you, jeh [15:07:19] milimetric: looks like one of the wm-bot processes was hung, did a clean stop/start on the service [15:07:45] thanks much [15:07:58] and thanks awight for pointing it out [15:13:45] elukey: it works great! [15:13:46] thank you! [15:15:50] ottomata: \o/ hope that it keeps working now, we'll see [15:16:03] !log re-enable timers on an-coord1001 after hive restart [15:16:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:24:32] 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Ottomata) [15:25:33] 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Ottomata) [15:25:35] 10Analytics: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10Neil_P._Quinn_WMF) >>! In T238560#5781792, @Nuria wrote: >> I disagree, and I would recommend that other analysts use Hue where appropriate. > I do not disagree. Hue is an OK tool for hadoop administrators... [15:25:38] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 9 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Ottomata) [15:31:31] 10Analytics, 10User-Elukey: CDH Jessie dependencies not available on Stretch - https://phabricator.wikimedia.org/T214364 (10MoritzMuehlenhoff) >>! In T214364#5774214, @elukey wrote: > I guess that libssl1.0.0 is part of either shlibs or misc Depends? I am not familiar about how they are populated, but I guess... [15:33:02] 10Analytics, 10Event-Platform, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242124 (10Ottomata) [15:33:54] 10Analytics, 10Event-Platform, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242126 (10Ottomata) [15:35:45] 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Ottomata) [15:37:53] 10Analytics, 10Event-Platform, 10SecTeam-Security-Reviews, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242124 (10chasemp) [15:45:04] 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Patch-For-Review, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Ottomata) [15:46:41] 10Analytics, 10Event-Platform: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242126 (10chasemp) [15:46:43] 10Analytics, 10Event-Platform, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242124 (10chasemp) [15:49:20] 10Analytics: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10Ottomata) I //think// Nuria means anything that stores user and/or dataset metadata in a separate database (other than the Hive Metastore). Hue and Superset both do this. Hue could be problematic lots of... [15:50:19] 10Analytics: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10mpopov) >>! In T238560#5782185, @Neil_P._Quinn_WMF wrote: >>>! In T238560#5781792, @Nuria wrote: >>> I disagree, and I would recommend that other analysts use Hue where appropriate. >> I do not disagree. Hu... [15:51:43] 10Analytics, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242125 (10chasemp) [15:52:26] 10Analytics, 10Event-Platform, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242126 (10chasemp) [15:53:20] 10Analytics, 10Cite, 10Reference Previews, 10CPT Initiatives (Modern Event Platform (TEC2)): Remove or simplify tracking metrics - https://phabricator.wikimedia.org/T242127 (10awight) [15:54:34] 10Analytics, 10Event-Platform, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242124 (10chasemp) [15:54:36] 10Analytics, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242125 (10chasemp) [15:54:38] 10Analytics, 10Event-Platform, 10Security-Team-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242126 (10chasemp) [15:58:01] elukey: ah! [15:58:17] can you help me fix analytics vlan rule for schema.svc ? [15:58:23] and schema.discovery.wmnet? [15:58:27] ottomata: 5 euros! [15:58:30] :D [15:58:30] port has changed [15:58:32] now on https [15:58:33] AH [15:58:39] you swindler. [15:58:40] i accept [15:58:42] ahahhaah [15:58:45] what is the change? [15:59:05] i think currnetly there is a rule for schema.svc.{eqiad,codfw}.wmnet [15:59:12] port 8190 [15:59:15] we should change to port 443 [15:59:20] and also add schema.discovery.wmnet while we are at it [15:59:38] hm which i guess willb e the same IPs [15:59:43] maybe nothing to do for discovery [16:00:52] for discovery I think that we are good since the eqiad/codfw ips are whitelisted [16:00:59] ya [16:01:00] if you are ok I'll replace t he port now [16:01:05] yup yes please! [16:10:06] ottomata: done! [16:10:09] 10Analytics, 10Operations, 10Product-Analytics, 10SRE-Access-Requests: Requesting access to analytics infrastructure - https://phabricator.wikimedia.org/T242026 (10SNowick_WMF) Hi @Nuria, I already have access to stats and notebooks, sorry for the confusion, this ticket has all the info for that completed... [16:14:19] 10Analytics, 10Operations, 10Product-Analytics, 10SRE-Access-Requests: Requesting access to analytics infrastructure - https://phabricator.wikimedia.org/T242026 (10Nuria) @SNowick_WMF Are you just missing access to ldap? can you access https://turnilo.wikimedia.org? [16:14:41] thank you luca! [16:15:37] np! [16:16:07] also TIL swindler :D [16:16:23] haha [16:24:55] 10Analytics: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10Nuria) Rephrasing: "I would strongly recommend analysts not to use hue at all" to do data science, that is. >I think Nuria means anything that stores user and/or dataset metadata in a separate database (... [16:28:15] ottomata: quick question regarding spark in notebooks on swap. using the 'pyspark - yarn' kernel, I now get errormessage "NameError: name 'spark' is not defined" suggesting that spark session is not automatically created anymore (I used kinit). previously typing 'spark' would just show the version etc. did anything change on swap for using spark? [16:28:40] hm [16:28:46] not that I know of! [16:28:53] maybe it just takes a bit longer for the session to be created? [16:30:01] trying too [16:30:28] mgerlach: i just got it to work, just took 10-20 seconds before it was ready [16:30:28] ottomata: strangely it works now ; ) maybe has to do with too many open notebooks? [16:30:47] mgerlach: it might have to do witih busy yarn cluster [16:30:54] 10Analytics, 10Event-Platform, 10Security-Readiness-Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242124 (10Jdforrester-WMF) [16:30:57] when starting a yarn shell, spark asks for resources on the cluster [16:30:57] 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Patch-For-Review, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Jdforrester-WMF) [16:30:58] ok thanks anyway [16:31:05] https://yarn.wikimedia.org/cluster/scheduler [16:31:09] doesen't look too busy atm [16:31:19] but it will still take some seconds at least for all that to happen [16:32:33] 10Analytics, 10Operations, 10Product-Analytics, 10SRE-Access-Requests: Requesting access to analytics infrastructure - https://phabricator.wikimedia.org/T242026 (10SNowick_WMF) I'm just not able to login to Hue, I can access Turnilo. [16:36:32] 10Analytics, 10Operations, 10Product-Analytics, 10SRE-Access-Requests: Requesting access to analytics infrastructure - https://phabricator.wikimedia.org/T242026 (10Nuria) What is the login you use to access turnilo? [16:38:38] 10Analytics, 10Operations, 10Product-Analytics, 10SRE-Access-Requests: Requesting access to analytics infrastructure - https://phabricator.wikimedia.org/T242026 (10SNowick_WMF) Shay Nowick [16:41:39] (03PS2) 10Fdans: (WIP) Add vue-i18n integration, English strings [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) [16:42:13] (03CR) 10jerkins-bot: [V: 04-1] (WIP) Add vue-i18n integration, English strings [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [16:43:20] (03CR) 10Nuria: [C: 03+2] "Tested number of distinct actors per hour for like a week and things looks good. So, merging." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/552943 (https://phabricator.wikimedia.org/T238360) (owner: 10Nuria) [16:43:26] (03CR) 10Nuria: [V: 03+2 C: 03+2] Table and workflow for features computations per actor per hour [analytics/refinery] - 10https://gerrit.wikimedia.org/r/552943 (https://phabricator.wikimedia.org/T238360) (owner: 10Nuria) [16:51:19] 10Analytics: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10MMiller_WMF) @elukey -- yes! It is working now. Thank you. [17:02:11] ping fdans [17:02:23] sorry! [17:12:42] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Prepare the Hadoop Analytics cluster for Kerberos - https://phabricator.wikimedia.org/T237269 (10elukey) [17:12:45] 10Analytics: Doubts and questions about Kerberos and Hadoop - https://phabricator.wikimedia.org/T238560 (10elukey) 05Open→03Resolved >>! In T238560#5782622, @MMiller_WMF wrote: > @elukey -- yes! It is working now. Thank you. Super, closing it again, let's open a new task if something comes up again (hopef... [17:46:46] any ideas, /etc/spark2/conf/spark-env.sh is not the same as puppet modules/cdh/templates/spark/spark-env.sh.erb. Any ideas where i should be looking? [17:48:06] The tl/dr is that the spark-env.sh we ship detects the python version of the local install to add the 3.5 or 3.7 python libraries (numpy, etc). I'm using deploy-mode cluster which means the python that spark-env.sh finds is not the python that will run the app, and it adds the wrong version (so i'm looking to fix it) [17:48:34] ebernhardson: I think that it is spark 1 vs spark 2 [17:48:57] cdh/spark/etc.. is spark 1 [17:49:06] elukey: ahh, is the other version entirely upstream? [17:49:27] ebernhardson: didn't get the question sorry [17:49:37] (/me slow now :) [17:49:50] the spark2 conf should be in puppet IIRC [17:49:50] ebernhardson: not upstream, but in our spark2 deb package [17:49:52] lemme check [17:49:57] ah no there you go :) [17:50:05] elukey: spark-env.sh adjusts the PYTHONPATH to include /usr/lib/spark2/python3.X. an-airflow has only 3.7, but our venv we ship only has 3.5. [17:50:05] spark-defaults is in puppet right [17:50:16] looking [17:50:18] or maybe in puppet... [17:50:20] errrr [17:50:30] hmm, why does airflow only have 3.7 [17:50:35] stat1005 has 3.5 [17:50:41] ottomata: newer debian version [17:50:43] maybe just needs installed explicitly [17:50:46] 3.5 packages are available [17:50:48] in buster [17:50:57] ottomata: in theory it shouldn't matter though, because we don't run the virtualenv on an-airflow [17:50:57] not that I remember [17:51:03] ottomata: are you sure? [17:51:16] ya python3.5 [17:51:21] ottomata: the problem is basically that in deploy-mode=cluster all python runs inside yarn containers, but we are still detecting the system python version that submitted the app to decide PYTHONPATH adjustments [17:51:35] ottomata: that is stretch though, buster ships with 3.7 [17:51:42] elukey@stat1004:~$ python3 --version [17:51:43] Python 3.5.3 [17:51:54] 1005 [17:52:07] i basically just want to add an environment variable to set the python version and skip detecting it [17:52:09] stat1005 was installed when buster was not yet final, so it may be possible that had python 3.5 [17:52:10] python3.5 --version [17:52:15] really? [17:52:34] yes do you remember that we had the same issue with jdk? [17:52:43] 8 was available on 1005 initially [17:52:58] hmmm [17:53:00] but then we got only 11 from upstream, and we had to backport [17:53:01] oh. [17:53:07] hmmMmmM ok wait. [17:53:07] hmmm [17:53:12] you are right sorry. [17:53:25] we should remove that from stat1005 rthen prob [17:53:39] yes yes you are completely right, it confuses things [17:53:43] doing it now [17:54:10] !log apt-get remove python3.5 on stat1005 [17:54:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:54:35] ok purged [17:54:52] ebernhardson: can you set PYSPARK_PYTHON to the one in your venv? [17:55:01] ottomata: no, because it doesn't exist on the host that submits the job [17:55:05] ottomata: the job is just a zip file in hdfs [17:55:16] hm [17:56:28] it looks like spark-env.sh is probably the upstream version, at least it ships in the .deb package [17:58:25] i could dupe it into puppet...but not sure that's ideal. Can file an upstream bug as well but that doesn't really solve anything soon [17:58:39] well, upstream is us :p [17:58:42] for packaging [17:58:49] oh, we package spark? didn't realize that [17:58:51] yup [17:59:15] ebernhardson: you are talking with the main developer! (Andrew) [17:59:19] i feel like i got this to work before [17:59:19] :D [17:59:22] trying to remember [17:59:47] ottomata: the last 4 lines of spark-env.sh basically say it all. It detects the local PYSPARK_PYTHON but doesn't know that spark is going to run in cluster deploy mode [17:59:51] yeah [17:59:58] ebernhardson what about [17:59:59] https://github.com/wikimedia/operations-debs-spark2/blob/debian/debian/README.Debian#L57-L58 [18:00:36] ottomata: doesn't help becaues the python path still gets set with the wrong python version, it's basically the python version of jarhell :) [18:00:49] only locally though, right? [18:00:58] no, that pythonpath is exported to the yarn containers [18:00:58] locally, it has to be 3.7, because 3.5 isn't available [18:01:04] not if you set [18:01:05] spark.executorEnv.PYTHONPATH [18:01:26] to the 3.5 one [18:01:36] it might be possible to do that without --archives [18:01:48] since /usr/lib/spark2/spark-2.4.4-python3.5.zip exists locally on all hadoop workers [18:01:49] not sure though [18:02:21] hmm, lemme try [18:04:30] (03CR) 10Nuria: [C: 03+1] "+2 if we have tested the job and code." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/556988 (https://phabricator.wikimedia.org/T236895) (owner: 10Ladsgroup) [18:06:26] hm but there is some issue about python loading binary deps out of zip files... [18:06:29] e.g. nuumpy [18:06:41] OH right, but --archives unzips it [18:07:50] (03CR) 10Ladsgroup: "> Patch Set 3: Code-Review+1" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/556988 (https://phabricator.wikimedia.org/T236895) (owner: 10Ladsgroup) [18:08:42] joal, :( the BlockMatrix that we were looking at was the one from Sparkmllib, but the one from criteo is different and has no multiply function... [18:09:24] I was looking into transforming it to a mllib BlockMatrix, by creating a converter, but it is very closed and not well documented [18:09:46] (03CR) 10Nuria: [C: 03+2] Fix WikidataArticlePlaceholderMetrics query [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/556988 (https://phabricator.wikimedia.org/T236895) (owner: 10Ladsgroup) [18:10:27] * elukey off! [18:13:29] joal, nuria, I think I'm going to go the other way and put everything inside the data quality job... [18:13:59] mforns: inside the one that consolidates the partititions? [18:14:03] If later we have another use case for the RSVD algorithm, we can consider factoring it out. [18:14:25] nuria, no no, the one that checks for anomalies [18:14:40] mforns: ah, like it was before? [18:14:45] yes [18:14:50] mforns: ok [18:14:55] ik [18:15:31] mforns: what i can do is do the comments if you want so you do not feel you have to over explain [18:15:49] nuria, I already have a good amount of comments [18:16:19] mforns: ok, will review [18:16:27] but feel free to add at review! [18:20:15] nuria, do you have 10 mins to discuss an idea to improve evaluation of an anomalous peak versus an anomalous drop? [18:21:11] mforns: sure, bc? [18:21:14] yep [18:27:13] 10Analytics, 10Event-Platform, 10Security Readiness Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242124 (10Jcross) p:05Triage→03Low [18:37:37] milimetric: merged reportupdater change [18:41:07] dentist appointment! backa in a bit [18:43:44] ottomata: hmm, i set spark.executorEnv.PYTHONPATH, but it's still getting the python3.7 adjustment overriding it :S [19:05:43] mforns: works for me :) [19:06:35] (03CR) 10Joal: Fix sqoop script and add CLI parameter (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/562325 (https://phabricator.wikimedia.org/T242015) (owner: 10Joal) [19:18:44] joal, ok [19:19:24] milimetric: if ok for you I'll update the patch above with Andrew suggestion so that it can be parft of tomorrow's train [19:23:49] (03CR) 10Milimetric: [C: 03+2] Fix sqoop script and add CLI parameter (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/562325 (https://phabricator.wikimedia.org/T242015) (owner: 10Joal) [19:27:13] (03PS2) 10Joal: Fix sqoop script and add CLI parameter [analytics/refinery] - 10https://gerrit.wikimedia.org/r/562325 (https://phabricator.wikimedia.org/T242015) [20:00:53] (03CR) 10Ottomata: [C: 03+1] Fix sqoop script and add CLI parameter [analytics/refinery] - 10https://gerrit.wikimedia.org/r/562325 (https://phabricator.wikimedia.org/T242015) (owner: 10Joal) [20:05:15] (03PS1) 10Milimetric: Encode to pagecounts-ez format with a UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/562597 (https://phabricator.wikimedia.org/T192474) [20:57:11] Gone for tonight team - see you tomorrow [21:00:20] laterrrrs [21:09:41] 10Analytics, 10Pywikibot: Provide some Pywikibot usage statistics for Python2.7 and Python3.x - https://phabricator.wikimedia.org/T242157 (10Multichill) [21:10:15] 10Analytics, 10Pywikibot: Provide some Pywikibot usage statistics for Python2.7 and Python3.x - https://phabricator.wikimedia.org/T242157 (10Multichill) [23:01:55] (03PS2) 10Mforns: [WIP] Add Spark/Scala module for anomaly detection [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/561674 (https://phabricator.wikimedia.org/T235486) [23:04:44] (03CR) 10Mforns: [C: 04-2] "Still missing proper comments and unit tests." (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/561674 (https://phabricator.wikimedia.org/T235486) (owner: 10Mforns) [23:05:21] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add Spark/Scala module for anomaly detection [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/561674 (https://phabricator.wikimedia.org/T235486) (owner: 10Mforns) [23:05:48] (03PS3) 10Mforns: [WIP] Add Spark/Scala module for anomaly detection [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/561674 (https://phabricator.wikimedia.org/T235486) [23:08:46] (03PS4) 10Mforns: [WIP] Add Spark/Scala module for anomaly detection [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/561674 (https://phabricator.wikimedia.org/T235486) [23:12:18] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add Spark/Scala module for anomaly detection [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/561674 (https://phabricator.wikimedia.org/T235486) (owner: 10Mforns) [23:13:31] (03PS5) 10Mforns: [WIP] Add Spark/Scala module for anomaly detection [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/561674 (https://phabricator.wikimedia.org/T235486)