[00:25:05] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10Krinkle) //("LTA" is a Wikimedia abbreviation for "long-term abuser".)// [03:13:19] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10Legoktm) @elukey do you know if SULWatcher is currently the only consumer of `#central`? If so, I think I can help port it to eventstreams once... [13:00:01] Hi team - here during siesta :) [13:33:35] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10revi) For the records: I don't use cvn-unification that often, but I always get pinged by someone who uses it whenever the bot is down (within... [13:55:39] (03CR) 10Joal: "Jobs are currently running using this code. We should deploy and restart them (refinery-source needed)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/565558 (https://phabricator.wikimedia.org/T238858) (owner: 10Joal) [14:02:22] (03PS4) 10Fdans: Add language selection functionality to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/564047 (https://phabricator.wikimedia.org/T238752) [14:02:34] (03CR) 10Fdans: "nuria: I've removed the hashes from the bundle file names. With this change only chunks have hashes. I don't see any reason to keep hashes" (0312 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [14:03:36] of course I applied the changes to the wrong patch ----- . ------ [14:03:47] o/ [14:04:02] #thingsthatwouldnthappenwithasanegitsystem [14:04:14] fdans: sorry I never showed you the missing metric thing yesterday, wanna hang out now and finalize the patch? [14:04:24] also, we should talk about the deploy /v2 strat [14:04:25] milimetric: yo I fixed that [14:04:28] oh! [14:04:29] nice [14:05:13] what'd you think of the deploy strat [14:08:55] (03PS16) 10Fdans: Add vue-i18n integration, English strings [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) [14:09:39] milimetric: reading [14:10:40] milimetric: what ottomata said about rsyncing was my original idea [14:11:02] milimetric: but what did yall resolve in the end? [14:11:09] what's in the email [14:11:29] (wikistats v2 saga, Andrew edition) [14:11:36] o wait [14:11:58] milimetric: maybe my zombie self at 4am read that email [14:13:23] :) I think we should ban brain activity except for between the hours of 10 and 20, I can't recall anything I read outside those hours and I'm afraid it's all sitting somewhere in my subconscious ready to revolt [14:15:17] who reads work emails outside of work hours? [14:16:40] ok milimetric ottomata let's do the 4 symlink solution then! [14:16:52] if the webpack stuff works, right? [14:16:55] does it work? [14:17:00] can you make it pack its stuff ina subdir? [14:17:09] ottomata: yea I think so [14:17:12] let me see [14:17:17] and actually, milimetric i was thikning, we might only want 2 symlinks, and do a full redirect from the v2/ urls, right? [14:17:18] fdans: I can work on that [14:17:23] just redirect v2/* to / [14:17:24] ? [14:17:25] you focus on the other patches [14:17:33] milimetric: hmmm [14:17:49] that way you don't have 2 canonical URLs for the same thing [14:17:55] ottomata: ok, sounds fine [14:17:58] milimetric: the thing is that the i18n patch comes with major changes to the webpack conf [14:18:00] yeah, makes sense [14:18:14] yeah, fdans I was gonna do it on top of that, we can land it today probably, right? [14:18:25] milimetric: I sure hope so [14:18:31] ok dokey, war room? [14:18:45] yea let's see how this internet does [14:21:57] 10Analytics, 10Event-Platform, 10Wikimedia-Extension-setup, 10Wikimedia-extension-review-queue: Deploy EventStreamConfig extension - https://phabricator.wikimedia.org/T242122 (10Ottomata) Bump! [14:22:05] 10Analytics, 10Event-Platform, 10Security Readiness Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242124 (10Ottomata) Bump! [14:23:55] 10Analytics, 10Event-Platform, 10Security Readiness Reviews: Security Review For EventStreamConfig extension - https://phabricator.wikimedia.org/T242124 (10Ottomata) Bump! One of our (O)KRs/goals this quarter is to deploy an EventLogging usage using Modern Event Platform components, including this one. Ther... [14:27:18] ebernhardson: did using python3.7 + a 3.7 venv for your spark stuff work? [14:28:15] Hi ottomata - not sure if you noticed yesterday - I have a working skein [14:39:57] fdans: I don't see the fix to the overlay status message when the metric doesn't work for all wikis [14:40:29] joal: when shall we hang today? [14:40:49] milimetric: now works for me! [14:41:00] ok let's batcave! [14:41:09] joining [14:50:20] joal: tell me! [14:50:23] i want to work on that today [14:51:50] ottomata: I documented my steps in https://phabricator.wikimedia.org/T243089 [14:53:15] hmmm [14:53:21] that is cool, but... [14:53:36] did you have the same skein problem I did? [14:53:39] ottomata: the issue I hit (I think it was the same you had) was due to unsetting https_proxy [14:53:41] i couldn't connect to the driver [14:53:45] OHHHHHHHHHH [14:53:47] interesting. [14:53:55] ok trying. [14:55:37] YES THANK YOU [14:55:43] :) [14:56:01] (03CR) 10Nuria: [C: 04-1] "Per e-mail conversation, we cannot remove hash from main bundle as then we would not be able to deploy anew version as new and old are in" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [14:56:29] nuria: submitting patch using semver [14:56:30] hi teammmm [14:56:40] fdans: semver? [14:57:19] nuria: instead of using a hash, using something like main.build.en.2.10.4.js [14:57:36] that way it is consistent across bundles [14:58:11] fdans: and how is that going to work for the non-main bundles? [14:58:54] nuria: the non-main bundles are not imported from index.html [14:59:41] so it's ok that they use their hash, because the value is embedded by webpack on the main bundle [14:59:42] fdans: ya, but we have two versioning schemes then, one for main bundles and one for subsequent bundles, while a possibility we might need to consider it does not seem the best option [15:00:24] fdans: i will be in meetings for the next couple hours but we can talk about this later on today [15:01:04] hmm the hash at the name on the file was never meant to be a form of versioning though [15:03:49] fdans: right, it is a means to deploy new code [15:04:48] fdans: hm, lots of sublocales: https://www.science.co.il/language/Locale-codes.php [15:07:39] fdans: we could add the version to all bundles? [15:13:02] joal: did you have any trouble with kerberos and skein? [15:13:05] i have kinit-ed [15:13:07] but am getting [15:13:13] Error: Kerberos ticket not found, please kinit and restart [15:14:19] AHH maybe jus trestaring [15:14:21] ok ok nm [15:14:28] this skein server in the backround thing is confusing [15:20:49] (03CR) 10Milimetric: "code looks good, only one comment, but looks like we'll have to resolve the bundle naming with nuria before we can merge either of these." (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/564047 (https://phabricator.wikimedia.org/T238752) (owner: 10Fdans) [15:32:39] milimetric , ottomata : did you guys got a task together for recruiting? [15:41:03] nuria: yes see email 'coding task for CPT hire' [15:42:41] milimetric: I mean there are lots of sublocales but not in translatewiki/datefns/numeral [15:53:20] ottomata: k [15:54:29] ottomata: nice! will review [15:59:12] ottomata: i still haven't had a chance to do the 3.7 test deploy, will soon-ish [16:05:15] k [16:29:13] ottomata, milimetric : i added a few things, please take a look [16:29:56] ottomata, milimetric : super good work [16:31:50] commented nuria [16:38:01] joal: if you have a min i could use a brain bounce on yarnspawner stuff [16:42:00] "wikistats v2 saga, Andrew edition" JAJAJA [16:51:14] hehehh [16:53:37] heya ottomata - was gone for kids [16:53:48] ottomata: pre-standup? [16:53:56] hm - probably in ops-sync [16:53:58] hey i think i got it. [16:54:08] it is a hadoop proxyuser + kerberos issue [16:54:12] goign to try to change configs on test cluster [16:54:16] thanks joal [16:54:23] humf [16:54:25] :) [16:56:44] 10Analytics, 10Security-Team, 10SecTeam Discussion, 10User-Elukey: VPN access to superset/turnilo instead of LDAP - https://phabricator.wikimedia.org/T242998 (10chasemp) [17:00:35] a-team, need to go to another meeting now, sent an e-scrum [17:00:50] np mforns [17:01:22] btw joal it is the data party meeting, in case you want to come (now or later on) [17:01:35] joal, https://meet.google.com/yfu-yywg-iic [17:01:38] Thanks mforns [17:02:37] milimetric: will you join us in wikimedia-research channel for the office hour? [17:02:44] milimetric: hiiii. :) [17:17:30] 10Analytics: Mediawiki history documentation for public dataset release - https://phabricator.wikimedia.org/T243426 (10Nuria) [17:17:51] 10Analytics, 10Analytics-Kanban: Mediawiki history documentation for public dataset release - https://phabricator.wikimedia.org/T243426 (10Nuria) a:03JAllemandou [17:18:58] 10Analytics, 10Analytics-Kanban: Mediawiki history public release: tsv format needs to quote every field - https://phabricator.wikimedia.org/T243427 (10Nuria) [17:49:52] (03PS17) 10Fdans: Add vue-i18n integration, English strings [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) [17:55:52] (03CR) 10Nuria: [C: 03+2] Enforce distinct revision in xml-dumps converter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/565554 (https://phabricator.wikimedia.org/T238858) (owner: 10Joal) [17:59:48] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10Masumrezarock100) **STRONG OPPOSE** for now - until we migrate SULWatcher bot's system to use EventStreams/Kafka. #cvn-unification channel is h... [18:01:26] (03Merged) 10jenkins-bot: Enforce distinct revision in xml-dumps converter [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/565554 (https://phabricator.wikimedia.org/T238858) (owner: 10Joal) [18:19:16] nuria: Not exactly sure where we'll be bc Golden Gate Park is rather large, but I found a few restaurants that are good at accommodating large groups and not too expensive. lmk what you think [18:19:17] - Pacific Catch (Seafood, East Side): https://pacificcatch.com [18:19:17] - Nopalito (Mexican, East Side): http://www.nopalitosf.com [18:19:17] - Cha Cha Cha (Caribbean, East Side): http://chachachasf.com [18:19:17] - Bambinos Ristorante (Italian, East Side): http://bambinosristorante.com [18:19:17] - Beach Chalet (American, West Side): https://www.beachchalet.com [18:22:11] ok joal back in kerberos land, maybe do need brain bounce [18:22:29] i'm getting the usual GSSException stuff, but afaict everything is authenticated just right, its just in the application itself once it runs that it fails [18:22:33] i only see that in the applicaiton logs [18:22:52] hm [18:23:00] ottomata: hdfs related? [18:23:05] hmmmm [18:23:28] ottomata: there are various tickets in hadoop-land once you have been accepted (thanks to kerberos) [18:23:28] don't think so [18:23:29] GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "analytics1028/10.64.36.128"; destination host is: "analytics1028.eqiad.wmnet":8032; [18:23:52] indeed [18:24:09] seems realy kerberos related, not hadoop internal [18:24:11] that's when trying to talk to the RM i guess [18:24:18] so the app master is trying to talk to RM [18:24:19] and failling [18:24:47] weird - This has worked for me (skein only, skein+dask) [18:25:09] Should we try that first ? [18:27:01] hm, you know i think i got a simple one working, hmmm [18:27:02] yes [18:34:17] ok so def simple hello world example works from just my user [18:34:22] with my cached kinit creds [18:34:27] good [18:34:28] now trying with proxyuser...if i can [18:34:41] trying to figure outhohw i'd do that [18:34:46] run skein driver as presto [18:34:52] but applicaiton submit using my user? [18:34:52] Meh? [18:35:02] can we batcave so that I better understand? [18:35:05] (i'm using presto user because it already ahs hadoop proxyuser rules) [18:35:06] ok [18:38:07] leila: omg I'm so sorry, I was triple booked and forgot! [19:12:39] 10Analytics, 10Analytics-Kanban: The guava error still persists in data quality bundles - https://phabricator.wikimedia.org/T241375 (10JAllemandou) I suggested in an email using a parquet table to handle DataQuality values, and I think that would indeed help. To read value-separated text file Hive uses `org.ap... [19:14:46] 10Analytics, 10Analytics-Kanban: The guava error still persists in data quality bundles - https://phabricator.wikimedia.org/T241375 (10mforns) Thanks for the explanation Joseph. Working on this right now. [19:15:33] milimetric ottomata hmmm can we do the symlink index.html => v8-assets/index.html [19:15:55] ok team - leaving for now - diner [19:15:58] putting the assets in a different place than index.html is proving a little tricky [19:25:04] milimetric: yeah. I had a look at your calendar and I figured something like that has happened. ;) [19:25:24] 10Analytics, 10Analytics-Kanban: Mediawiki history public release: tsv format is not correctly parsable - https://phabricator.wikimedia.org/T243427 (10JAllemandou) [19:38:50] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10Xqt) Pywikibot has a powerfull eventstreams handler. Probably we can use it or its code parts for this proposal. [19:40:29] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10Ottomata) > until we migrate SULWatcher bot's system to use EventStreams/Kafka ... Can we do that? [20:36:24] haha joal what a world. things are kind of working but anaconda is very confusing in a yarn container [20:36:27] especially [20:36:36] somehow my python exectubles in scripts are [20:36:37] e.g. [20:36:42] #!/var/lib/hadoop/data/l/yarn/local/usercache/otto/appcache/application_1576771377404_31159/container_e05_1576771377404_31159_01_000001/environment/bin/python [20:36:46] but guess what... [20:36:55] LINUX bash shebang lenght limit is 128 chars!!! [20:37:00] WHY would linux do that ?! [20:40:50] 10Analytics, 10Tool-stewardbots, 10User-Elukey: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712 (10MarcoAurelio) >>! In T242712#5824196, @Ottomata wrote: >> until we migrate SULWatcher bot's system to use EventStreams/Kafka > ... Can we do th... [20:46:46] ottomata: man - that's tricky!!! [20:46:55] ottomata: back to good ol' venv? [20:47:10] it would have the same problem [20:47:24] i'm actually not sure how that path got changed, i need to understand what conda activate does [20:47:39] there is a nice way to layer/stack conda envs [20:48:08] i was trying to see if i could do a globally installed one (via a custom .deb maybe? or loaded in hdfs) and then user local ones [20:48:18] problem is, the user local envs will be in the yarn appcache temp dir. [20:48:22] if they are running in yarnspawner [20:48:52] which will be deleted if the user's jupyter notebookservice yarn app is killed [20:49:17] i got it working so that the .ipynb filies are saved in hdfs! [20:49:20] so that's pretty cool! [20:49:35] unfortunetly all the other fs interactions are in the worker node fs [20:49:57] the jupyter ContentsMananger API doesn't abstract that stuff., just things that jupyter itself saves [20:51:04] i think to do this the way I really want; we'd need a writable shared reulgar filesystem, not HDFS blabla [20:51:13] if it worked, fuse mount would work [20:51:14] hm [20:51:33] maybe in 2020 there are better hdfs fs stuff... [20:54:33] HMm, NFS is built into HDFS??? [20:54:35] https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html [21:02:56] I didn't know that one ottomata (NFS) [21:03:12] me neither [21:03:19] there are a lot of fs libs out there actually [21:03:25] we just always used the one from cloudera [21:03:29] i bet we could find a good one... [21:03:49] ottomata: the reason I think venv-pack is better for the shebang issue is that it uses the regular python (defined at venv creation), therefore /usr/bin/python3 for isntance - Less than 128 chars :) [21:04:55] right [21:05:01] but then that exact python has to be installed everywhere [21:05:13] ottomata: which it is (I think) [21:05:18] not for anaconda [21:05:22] ottomata: true [21:05:30] anaconda is a big analytics based python distro that uses conda [21:05:31] You can pick the exact python version you wish [21:05:57] it prepackages tons of python packages and resolves lots of issues we've run into with e.g. binary deps [21:06:59] anyway am just trying things, maybe venv is better but from what I can tell conda does a much better job at creating re-usable envs [21:07:25] i'm trying to see if i can use YARN+HDFS to make a 100% thin jupyterhub client [21:07:41] so we never have to deal with stat boxes or notebook client compute nodes [21:08:00] if we can make it easy and fast and transparent for folks to just just use jupyter in hadoop like they do locally [21:08:06] it would make our maintainence lives much easier [21:08:07] but [21:08:16] i think i can't do it without a good hdfs fs mount [21:08:22] it needs a shared filesystem [21:08:35] i thought i could with https://jcrist.github.io/hdfscm/ [21:09:06] For what I read from you earlier, it feels like Jupyter Server needs a regular posix compatible FS, not a special one (hdfs) [21:09:38] right [21:09:41] it doesn't work hdfscm? [21:09:41] exactly [21:09:47] it does work, but [21:09:58] the ContentsManager only is for stuff done directly by jupyter notebook server [21:10:02] not stuff by users [21:10:09] e.g., if I do !pip install ... [21:10:14] that is a shell command [21:11:02] i.e. only files interacted with via the jupyter notebook server GUI (like new notebook .ipynb files) use the CM [21:11:39] the environments running code, packages etc are stored on the local FS of the container - is that? [21:11:48] that's cool and good, and means that if my yarnspawner based notebook server dies and is respawned on a new worker node; my notebooks will still be there (in my hdfs user dir) [21:12:10] but, any data files I manually created will be lost once the app clears the nodemanager tempdir [21:12:24] yeah, i have the conda packed env in hdfs [21:12:37] and yarnspawner will auto unpack it and source it [21:12:46] and run python from it [21:12:48] any update to the conda-packed (which is unpacked locally) is lost [21:12:55] kinda like we do with spark --archives, (just with deps) [21:12:59] up [21:13:04] +y [21:13:25] yeah, and actually, the unpacked stuff is not writable by my user, its only writable by yarn :p [21:13:42] so i'd need a secondary environment to install stuff in anyway [21:13:53] which is probably good; the way we do it in SWAP now is bad [21:14:12] currently we COPY the starting venv to the users home dir when the user logs in for the first time. [21:14:57] which is totally unmaintainable. To upgrade any packages we have to loop thhrough EVERY user venv and upgrade, and hope the user (or us) hasn't messed anything up [21:15:30] yeah [21:15:43] I don't know how the "secondary env" would work though [21:15:58] Like a global/local env I assume [21:18:53] yeah [21:18:59] it looks like conda can 'nest' them somehow [21:19:08] https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#nested-activation [21:20:56] ottomata: My understanding of the 'nesting' is that it wou;d allow to change from one env to another, not to have a base one (global) with a set of preinstalled deps and another one (local) with updated deps [21:21:27] joal: conda activate --stack myenv [21:21:27] ? [21:22:23] Ah right - I wonder if that would work for deps [21:25:53] yeah me too [21:26:00] i need to explore that locally [21:26:04] got a lot more to figure out now [21:26:09] thanks for helping me get yarnspawner working [21:26:11] it actually works!!! [21:26:16] \o/ [21:33:06] joal: how did it go with your yarn dask experiements btw? [21:33:32] could/would it be possible to create a notebook kernel that set up running and distributing python stuff in yarn via that? [21:33:41] 'python in yarn without spark' ? :) [21:34:04] ottomata: test went well :) [21:34:30] ottomata: I indeed think we can easily provide Dask on the cluster :) [21:35:22] ottomata: we'll have the same issue as with spark before dynamic allocation: resource fight might become something more proeminent, but except from that it's all positive :) [21:35:57] hm aye [21:36:10] i'll add this to things to explroe for newpyter then too [21:36:18] could be cool to have some kernel stuff built in that makes that easy [21:37:15] ottomata: I read some doc also on Dask and what it allows - Interestingly it is pretty similar to spark in term of what it allows for ML, except for a specific class of algorithm that can be "incrementaly updated" [21:37:44] oh? [21:38:19] That is: you can distribute compute power over the grid when data is small, you can use dask-predifined ML algo when data is big, or you can use scikit-learn "incrementaly updatable" algos [21:38:39] The thing spark doesn't do is the last [21:40:18] Also the fact that dask and scikit are integrated should facilitate a lot I guess :) [21:55:56] hm cool [22:24:30] (03PS1) 10Joal: Update mediwiki-history dumper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/566609 (https://phabricator.wikimedia.org/T243427)