[01:18:10] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience, 07Epic: AB Test TTS For You Analysis - https://phabricator.wikimedia.org/T435669#12542805 (10SNowick_WMF) TTS Analysis shared with @HNordeenWMF - will review and summarize later this week, please message me if you want access. [07:54:44] 06Machine-Learning-Team (Q2 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2026-27 Text-to-Speech V2 Experiment - https://phabricator.wikimedia.org/T441938#12543283 (10kevinbazira) [08:23:56] (03PS1) 10Kevin Bazira: tts-section-generator: upgrade to trixie for ffmpeg 7.1.5 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368609 (https://phabricator.wikimedia.org/T444440) [08:54:15] (03CR) 10Bartosz Wójtowicz: [C:03+1] tts-section-generator: upgrade to trixie for ffmpeg 7.1.5 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368609 (https://phabricator.wikimedia.org/T444440) (owner: 10Kevin Bazira) [09:01:21] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: upgrade to trixie for ffmpeg 7.1.5 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368609 (https://phabricator.wikimedia.org/T444440) (owner: 10Kevin Bazira) [09:04:23] (03Merged) 10jenkins-bot: tts-section-generator: upgrade to trixie for ffmpeg 7.1.5 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368609 (https://phabricator.wikimedia.org/T444440) (owner: 10Kevin Bazira) [09:25:40] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27): Upgrade vLLM in AMD ROCm base image to v0.32.0 - https://phabricator.wikimedia.org/T442138#12543514 (10BWojtowicz-WMF) Thank you Aiko, I'm leaving this how-to guide for the future! #### Upgrading the vLLM base image (AMD ROCm) **1. Prototype on ML-Lab** *... [10:58:51] 06Machine-Learning-Team (Q2 FY2026-27): inference-services: trigger image builds when python/requirements.txt changes - https://phabricator.wikimedia.org/T444292#12543876 (10gkyziridis) [10:59:01] 06Machine-Learning-Team (Q2 FY2026-27): inference-services: trigger image builds when python/requirements.txt changes - https://phabricator.wikimedia.org/T444292#12543877 (10gkyziridis) 05Open→03Resolved [11:05:13] (03PS1) 10Gkyziridis: python: upgrade aiohttp to 3.14.5 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368637 [11:16:21] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27): Upgrade vLLM in AMD ROCm base image to v0.32.0 - https://phabricator.wikimedia.org/T442138#12543973 (10BWojtowicz-WMF) I'm resolving the task and will open up a follow-up task for migration. Here are some notes from experimentation and testing of the new vl... [11:16:42] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27): Upgrade vLLM in AMD ROCm base image to v0.32.0 - https://phabricator.wikimedia.org/T442138#12543987 (10BWojtowicz-WMF) 05Open→03Resolved [11:17:59] 06Machine-Learning-Team (Q2 FY2026-27): Migrate LLM services to the new vLLM image - https://phabricator.wikimedia.org/T444464 (10BWojtowicz-WMF) 03NEW [11:21:34] (03CR) 10CI reject: [V:04-1] python: upgrade aiohttp to 3.14.5 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368637 (owner: 10Gkyziridis) [11:38:43] (03PS2) 10Gkyziridis: python: upgrade aiohttp to 3.14.5 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368637 [11:44:12] 06Machine-Learning-Team (Q2 FY2026-27): Migrate llm-qwen3-14b GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T444469 (10gkyziridis) 03NEW [11:57:19] (03CR) 10CI reject: [V:04-1] python: upgrade aiohttp to 3.14.5 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368637 (owner: 10Gkyziridis) [12:05:50] (03CR) 10Gkyziridis: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368637 (owner: 10Gkyziridis) [12:11:10] 06Machine-Learning-Team (Q2 FY2026-27): Migrate LiftWing LLMs to LLMInferenceService - https://phabricator.wikimedia.org/T441270#12544341 (10BWojtowicz-WMF) Resolving the task with the final update on the migration. Migrated to LLMInferenceService: * llm-qwen3-14b - T441274 * llm-qwen36-27b - T441644 * llm-qwen... [12:15:28] 06Machine-Learning-Team (Q2 FY2026-27): Clean up unused inference-services code after LLMISVC migration - https://phabricator.wikimedia.org/T444471 (10BWojtowicz-WMF) 03NEW [12:16:51] 06Machine-Learning-Team (Q2 FY2026-27): Migrate LiftWing LLMs to LLMInferenceService - https://phabricator.wikimedia.org/T441270#12544384 (10BWojtowicz-WMF) 05Open→03Resolved [12:44:50] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate LLM services to the new vLLM image - https://phabricator.wikimedia.org/T444464#12544531 (10BWojtowicz-WMF) 05Open→03In progress a:03BWojtowicz-WMF [13:21:03] (03CR) 10Gkyziridis: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368637 (owner: 10Gkyziridis) [13:43:24] (03CR) 10Gkyziridis: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1368637 (owner: 10Gkyziridis) [13:46:27] 06Machine-Learning-Team (Q2 FY2026-27): Predictions an Opensource Pangram model - https://phabricator.wikimedia.org/T444488 (10OKarakaya-WMF) 03NEW [13:49:16] 06Machine-Learning-Team (Q2 FY2026-27): Predictions from an Opensource Pangram model - https://phabricator.wikimedia.org/T444488#12544874 (10OKarakaya-WMF) [13:54:34] (03CR) 10AikoChou: "Hi Georgios, thanks for this!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359881 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [14:36:02] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27): Upgrade vLLM in AMD ROCm base image to v0.32.0 - https://phabricator.wikimedia.org/T442138#12545145 (10isarantopoulos) The documentation described in {T442138#12543514} has been added by Bartosz to the following wikitech page https://wikitech.wikimedia.o... [14:37:04] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Clean up unused inference-services code after LLMISVC migration - https://phabricator.wikimedia.org/T444471#12545163 (10isarantopoulos) [15:09:34] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate llm-qwen3-14b GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T444469#12545394 (10gkyziridis) Deployed ` $ kube_env llm ml-serve-eqiad $ kubectl get pods -o wide | grep qwen3-14b llm-qwen3-14b-kserve-69b95d5479... [15:09:42] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate llm-qwen3-14b GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T444469#12545395 (10gkyziridis) 05Open→03Resolved [15:10:11] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate ml-services GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T441680#12545397 (10gkyziridis) [15:37:59] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work, 13Patch-For-Review: Clean up unused inference-services code after LLMISVC migration - https://phabricator.wikimedia.org/T444471#12545567 (10gkyziridis) I am starting by deleting the CI/CD pipelines in `zuul/layout` and `jjb/project-pipelines`