[08:32:26] <isaranto>	 hello!
[08:40:34] <isaranto>	 it is interesting that the only CI tests that are failing in the optimum benchmark are the ones related to pytorch+rocm https://github.com/huggingface/optimum-benchmark?tab=readme-ov-file#cli-
[08:40:35] <isaranto>	 :D
[09:10:24] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team, 07OKR-Work: Create event stream for article-country model-server hosted on LiftWing - https://phabricator.wikimedia.org/T382295#10414984 (10isarantopoulos) @Isaac Is there an expected timeline for when this needs to be ingested? The team doesn’t have the capacity to take...
[09:23:57] <wikibugs>	 (03CR) 10Nik Gkountas: [C:03+2] Randomize collection-based recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1105440 (https://phabricator.wikimedia.org/T381888) (owner: 10Sbisson)
[09:25:25] <wikibugs>	 (03Merged) 10jenkins-bot: Randomize collection-based recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1105440 (https://phabricator.wikimedia.org/T381888) (owner: 10Sbisson)
[13:39:56] <kevinbazira>	 o/ I've refactored the hf optimum benchmark automation tool to use cli instead of a python sdk to easily support running multiple benchmarks:
[13:39:56] <kevinbazira>	 https://gitlab.wikimedia.org/repos/machine-learning/huggingface-optimum-benchmark-automation
[13:39:56] <kevinbazira>	 try it and let me know what you think :)
[13:51:49] <isaranto>	 kevinbazira: nice, I will try to run it either today or Monday
[13:53:00] <isaranto>	 I would assume that the same options would be available through the python sdk as well but if not ¯\_(ツ)_/¯
[13:53:33] <isaranto>	 I mean I was wondering "doesnt the cli tool just use the python sdk underneath?"
[13:53:58] <isaranto>	 anyway, just laying out my thoughts. thanks for working on this!
[13:55:42] <kevinbazira>	 isaranto: the multirun option is not in the python sdk. the optimum-benchmark cli relies on hydra for that: https://hydra.cc/docs/tutorials/basic/your_first_app/simple_cli/
[13:57:02] <isaranto>	 ok.. I like hydra a lot!
[13:57:44] <kevinbazira>	 TIL: about hydra and it's super cool!
[13:57:53] <isaranto>	 but can you set the # of input/output tokens in the python sdk? if yes, we could use that then to create our own multirun setup in python
[13:58:17] <isaranto>	 the benefit would be that we can manipulate the output of the benchmark and produce the plots that we want as well 
[13:58:46] <isaranto>	 I'll try to run it first  and come back with more questions probably :)
[14:02:16] <kevinbazira>	 okok the cli setup is much cleaner. we use `.yaml` files to set the benchmark configs: https://gitlab.wikimedia.org/repos/machine-learning/huggingface-optimum-benchmark-automation/-/tree/main/configs 
[16:02:58] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team, 07OKR-Work: Create event stream for article-country model-server hosted on LiftWing - https://phabricator.wikimedia.org/T382295#10416130 (10Isaac) > Is there an expected timeline for when this needs to be ingested? The team doesn’t have the capacity to take this on right...
[17:11:52] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team, 07OKR-Work: Create event stream for article-country model-server hosted on LiftWing - https://phabricator.wikimedia.org/T382295#10416607 (10Isaac) Adding @EBernhardson as well because when this stream moves forward, we'll want to ingest it into Search (past conversations...