[00:04:39] <wikibugs>	 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10drafttopic-modeling, 10Machine-Learning-Team (Active Tasks): ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10Halfak) Looks like I was mistaken while I was reading the graphs.  The change is from 27% to 22% of //a...
[10:27:59] * elukey afk! lunch
[13:35:23] <wikibugs>	 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10drafttopic-modeling, 10Machine-Learning-Team (Active Tasks): ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10elukey) On paper we should have free memory available on Production nodes, but ideally the three change...
[17:02:14] <wikibugs>	 10Jade, 10Beta-Cluster-Infrastructure, 10MediaWiki-ContentHandler, 10Patch-For-Review, and 2 others: Beta cluster: The content model 'JadeJudgment' is not registered - https://phabricator.wikimedia.org/T247476 (10DannyS712) >>! In T247476#7047673, @Ladsgroup wrote: > I can't reproduce the error anymore giv...
[17:37:35] <elukey>	 theo*fpa gave me this AWESOME example for Feast, that I am still digesting but I think it makes mroe sense now 
[17:37:38] <elukey>	 https://github.com/feast-dev/feast/blob/master/examples/minimal/minimal_ride_hailing.ipynb
[17:38:37] <elukey>	 so the so called "Feature Registry" can be saved into multiple formats, from files to dbs I think, and it stores the metadata about the feature datasets
[17:39:35] <elukey>	 there is a part in the python notebook about using Spark 
[17:41:30] <elukey>	 now at some point the client does
[17:41:31] <elukey>	 job = client.get_historical_features
[17:42:00] <elukey>	 that internally leverages (IIUC) Pyspark to query data from anywhere, and put results in s3 or in our case, maybe hdfs?
[17:42:05] <elukey>	 (see read_parquet)
[17:43:50] <elukey>	 the main thing that worries me is where this code runs, since if this client is on TrainWing then adding Kerberos auth may become a nightmare
[17:44:34] <wikibugs>	 10Jade, 10Beta-Cluster-Infrastructure, 10MediaWiki-ContentHandler, 10Patch-For-Review, and 2 others: Beta cluster: The content model 'JadeJudgment' is not registered - https://phabricator.wikimedia.org/T247476 (10Ladsgroup) Is that error related to jade and its content model? It doesn't look like it TBH
[17:44:38] <elukey>	 (kerberos for kubernetes pods I mean, I don't think it works)
[17:46:43] <elukey>	 maybe we could run the get_historical_feature() from a python notebook on stat100x (kerberized etc..) and then push the result to S3 (the MOSS cluster)
[17:47:00] <elukey>	 then pods on trainwing could retrieve the data from it
[17:47:15] <elukey>	 but we'd loose granularity on data access
[17:47:22] <elukey>	 am I crazy?? probably
[17:48:23] <elukey>	 if I am right the good news is that notebooks on stat100x works really well :(
[17:48:26] <elukey>	 err :)
[17:49:06] <elukey>	 but the need to push a lot of data to S3/MOSS/Swift may be an issue
[17:49:15] <wikibugs>	 10Jade, 10Beta-Cluster-Infrastructure, 10MediaWiki-ContentHandler, 10Patch-For-Review, and 2 others: Beta cluster: The content model 'JadeJudgment' is not registered - https://phabricator.wikimedia.org/T247476 (10DannyS712) >>! In T247476#7049438, @Ladsgroup wrote: > Is that error related to jade and its c...
[17:49:21] <elukey>	 ideally we'd just pull the data from HDFS
[17:55:49] <elukey>	 https://engineering.linkedin.com/blog/2020/open-sourcing-kube2hadoop
[17:59:48] <elukey>	 ok mental sanity gone for the moment, will restart on monday :)
[17:59:52] <elukey>	 have a good weekend folks
[21:14:33] <wikibugs>	 10Jade, 10Machine-Learning-Team, 10Project-Admins: Archive Jade project - https://phabricator.wikimedia.org/T281430 (10calbon) Awesome, thanks Peachy88