[05:07:06] (03PS3) 10Kevin Bazira: decision-models: add clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) [05:30:20] (03PS2) 10Kevin Bazira: decision-models: containerize clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352492 (https://phabricator.wikimedia.org/T440223) [07:17:41] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12409848 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos [07:18:24] 06Machine-Learning-Team (Q2 FY2026-27), 10Add-Link-Structured-Task, 06Growth-Team: Incorporate user decision signals into Add Link model training - https://phabricator.wikimedia.org/T435831#12409852 (10isarantopoulos) [07:19:10] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27): Move llm isvcs to new partitions - https://phabricator.wikimedia.org/T439555#12409853 (10isarantopoulos) [07:19:31] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work, 13Patch-For-Review: Add 'test/unit//' paths to Zuul trigger file filters - https://phabricator.wikimedia.org/T440288#12409854 (10isarantopoulos) [07:20:01] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12409855 (10isarantopoulos) [07:20:26] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: qwen3-14b model server: Return specific http status (400) when the prompt exceeds MAX_MODEL_LEN - https://phabricator.wikimedia.org/T439967#12409858 (10isarantopoulos) [07:21:27] 06Machine-Learning-Team (Q2 FY2026-27), 06Reader Growth Team: [ML TTS] Expose structured sentence, paragraph, and revision metadata for Read-Along synchronization - https://phabricator.wikimedia.org/T438228#12409873 (10isarantopoulos) [07:23:41] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: liftwing-studio: Set up a shared local LibreChat environment - https://phabricator.wikimedia.org/T438673#12409875 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos This work has resulted in the setup available under the directory https... [07:25:17] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: ORESFetchScoreJobKafkaLag (1 alert) and LiftWingServiceErrorRate (1 alert) - https://phabricator.wikimedia.org/T436525#12409880 (10isarantopoulos) 05Open→03Resolved Resolving this for now as there was no follow up action and it has not occurred... [07:25:47] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12409886 (10isarantopoulos) 05Open→03Resolved [07:27:10] 06Machine-Learning-Team (Q2 FY2026-27), 06Discovery-Search: Create SLO for the jina-embeddings model server using Sloth - https://phabricator.wikimedia.org/T439245#12409891 (10isarantopoulos) a:03OKarakaya-WMF [07:30:22] 06Machine-Learning-Team (Q2 FY2026-27): Define and implement SLOs for the public LLM inference services - https://phabricator.wikimedia.org/T440307#12409900 (10isarantopoulos) WIP SLO draft is currently being worked on here https://wikitech.wikimedia.org/wiki/SLO/LiftWing_LLM [07:34:12] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Decide on the future shape of ML/LW SLOs, and how they are monitored - https://phabricator.wikimedia.org/T434525#12409916 (10isarantopoulos) We've started some work on the SLOs for the following services... [08:58:36] (03PS1) 10AikoChou: tool-server: Refactor wikipedia_semantic_search into search and formatting [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353033 (https://phabricator.wikimedia.org/T440444) [08:58:38] (03PS1) 10AikoChou: tool-server: Add card grid UI to wikipedia_semantic_search [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353034 (https://phabricator.wikimedia.org/T440444) [09:22:35] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work, 13Patch-For-Review: Add 'test/unit//' paths to Zuul trigger file filters - https://phabricator.wikimedia.org/T440288#12410272 (10achou) Thanks for the suggestion! I like the idea of keeping tests next to the model code, but there... [09:37:25] (03CR) 10Ozge: decision-models: add clef model-server (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [09:38:13] (03CR) 10Ozge: [C:03+1] decision-models: containerize clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352492 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [10:43:44] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12410594 (10achou) **Plan: ml-serve1015 baseline benchmark... [10:45:41] (03PS4) 10Kevin Bazira: decision-models: add clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) [10:49:28] (03CR) 10Kevin Bazira: decision-models: add clef model-server (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [10:52:01] (03CR) 10Ozge: [V:03+1 C:03+1] decision-models: add clef model-server (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [10:56:17] (03CR) 10Kevin Bazira: [C:03+2] decision-models: add clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [10:57:38] (03Merged) 10jenkins-bot: decision-models: add clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [11:02:21] (03CR) 10Kevin Bazira: [C:03+2] decision-models: containerize clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352492 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [11:02:49] (03Merged) 10jenkins-bot: decision-models: containerize clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352492 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [11:28:08] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work, 13Patch-For-Review: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12410847 (10achou) [11:32:35] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work, 13Patch-For-Review: Add 'test/unit//' paths to Zuul trigger file filters - https://phabricator.wikimedia.org/T440288#12410867 (10gkyziridis) Thank you @hashar for your comment. I suggest we stick on the current setup since this... [11:47:11] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Increase number of cpus in llm namespace. - https://phabricator.wikimedia.org/T440575 (10OKarakaya-WMF) 03NEW [11:49:37] 06Machine-Learning-Team (Q2 FY2026-27): Define and implement SLOs for the public LLM inference services - https://phabricator.wikimedia.org/T440307#12410971 (10gkyziridis) ==== Qwen3-14b Benchmark on ml-serve1013/1014 96Gb partitions. ==== These are the results of the vllm-bench //(latencies in ms)//. More inf... [11:59:09] (03PS1) 10Gkyziridis: benchmarks: add llm-qwen3-14b baseline (experimental ns) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353138 (https://phabricator.wikimedia.org/T440307) [11:59:36] (03CR) 10CI reject: [V:04-1] benchmarks: add llm-qwen3-14b baseline (experimental ns) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353138 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:01:12] (03PS2) 10Gkyziridis: benchmarks: add llm-qwen3-14b baseline (experimental ns) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353138 (https://phabricator.wikimedia.org/T440307) [12:04:59] (03CR) 10Kevin Bazira: [C:03+1] benchmarks: add llm-qwen3-14b baseline (experimental ns) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353138 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:07:04] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Define and implement SLOs for the public LLM inference services - https://phabricator.wikimedia.org/T440307#12411036 (10OKarakaya-WMF) I run the previous one on stat1010: https://phabricator.wikimedia.org/T431851#12145154 [12:07:20] (03CR) 10Gkyziridis: [C:03+2] benchmarks: add llm-qwen3-14b baseline (experimental ns) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353138 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:07:51] (03Merged) 10jenkins-bot: benchmarks: add llm-qwen3-14b baseline (experimental ns) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353138 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:32:41] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Update number of cpus in namespaces. - https://phabricator.wikimedia.org/T440575#12411135 (10OKarakaya-WMF) [12:37:47] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Update number of cpus in namespaces. - https://phabricator.wikimedia.org/T440575#12411157 (10OKarakaya-WMF) [12:58:16] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Update number of cpus in namespaces. - https://phabricator.wikimedia.org/T440575#12411275 (10OKarakaya-WMF) [12:59:00] 06Machine-Learning-Team (Q2 FY2026-27): Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588 (10OKarakaya-WMF) 03NEW [12:59:23] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411291 (10OKarakaya-WMF) a:03OKarakaya-WMF [13:03:23] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411312 (10OKarakaya-WMF) [13:03:36] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411326 (10OKarakaya-WMF) [13:03:47] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411328 (10OKarakaya-WMF) [13:23:16] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411388 (10OKarakaya-WMF) [13:25:14] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Define and implement SLOs for the public LLM inference services - https://phabricator.wikimedia.org/T440307#12411393 (10gkyziridis) ==== Qwen36-27b Benchmark on ml-serve1013/1014 96Gb partitions. ==== These are the results of the vllm-bench (latencies... [13:29:26] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411401 (10OKarakaya-WMF) [13:30:46] (03PS1) 10Gkyziridis: benchmarks: add llm-qwen36-27b on a 96GB partition (ml-serve1013/1014) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353202 (https://phabricator.wikimedia.org/T440307) [13:36:43] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411454 (10OKarakaya-WMF) [13:37:44] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411456 (10OKarakaya-WMF) [13:38:09] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411461 (10OKarakaya-WMF) [13:39:00] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12411463 (10OKarakaya-WMF) [13:49:30] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Update number of cpus in namespaces. - https://phabricator.wikimedia.org/T440575#12411523 (10OKarakaya-WMF) [17:29:08] 10Lift-Wing, 06Machine-Learning-Team, 06ServiceOps, 10ServiceOps-SharedInfra, and 2 others: Investigate X-Wikimedia-Debug routing for APIs going through the rest-gateway - https://phabricator.wikimedia.org/T428909#12412999 (10Scott_French) a:05Scott_French→03Clement_Goubert Transferring this back to @C... [20:54:16] 06Machine-Learning-Team: Lots of old data in ores_classification table - https://phabricator.wikimedia.org/T440358#12413913 (10Ladsgroup) The clean up script is really slow. I try to debug it. [21:07:36] (03PS1) 10Ladsgroup-claude: Fix two bugs on PurgeScoreCache [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1353433 (https://phabricator.wikimedia.org/T440358) [21:29:30] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Oct 07 - Oct 20]): TTS MVP within "For you" Data Validation - https://phabricator.wikimedia.org/T438506#12414003 (10SNowick_WMF) 05Open→03Resolved