[07:33:22] (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you for the answers and fixes, LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360378 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [07:37:27] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: split long sections across multiple requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360378 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [07:44:57] (03Merged) 10jenkins-bot: tts-section-generator: split long sections across multiple requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360378 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [08:51:43] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 06Research, 10Tools: FactDiff: an editor worklist of conflicting dates and numbers across Wikipedia language editions (WIP, feedback welcome) - https://phabricator.wikimedia.org/T441279#12493863 (10isarantopoulos) [09:37:12] 06Machine-Learning-Team (Q2 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12494083 (10isarantopoulos) [09:37:15] 06Machine-Learning-Team (Q2 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12494084 (10isarantopoulos) [09:44:41] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12494120 (10OKarakaya-WMF) sure, I'll update with LLMisvc and remove sclae-to-zero. [10:26:11] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Add the ML team to the POSIX group `docker` on `ml-build1001` - https://phabricator.wikimedia.org/T393566#12494299 (10Gehel) [10:26:27] 06Machine-Learning-Team (Q2 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12494305 (10Gehel) [10:26:35] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12494307 (10Gehel) [10:26:41] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Decide on the future shape of ML/LW SLOs, and how they are monitored - https://phabricator.wikimedia.org/T434525#12494309 (10Gehel) [10:29:01] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-30 - 2026-11-20): ml-lab: Update pytorch-rocm to a current ROCm-compatible release - https://phabricator.wikimedia.org/T428023#12494370 (10Gehel) [10:29:07] 06Machine-Learning-Team, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Document procedure for updating the underlying distro for model servers (e.g. Bullseye -> Bookworm) - https://phabricator.wikimedia.org/T352189#12494381 (10Gehel) [10:29:43] 06Machine-Learning-Team (Q1 FY2026-27), 10Prod-Kubernetes, 06ServiceOps, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), and 3 others: Update istio-cni to 1.29 - https://phabricator.wikimedia.org/T434198#12494408 (10Gehel) [10:33:05] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12494461 (10Gehel) [10:33:47] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12494480 (10Gehel) [11:33:00] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Kubernetes, 13Patch-For-Review: Extend KServe's LLMInferenceService by Gateway API Inference Extension and KEDA autoscaling - https://phabricator.wikimedia.org/T436650#12494681 (10Gehel) [11:35:03] 06Machine-Learning-Team, 10Cassandra, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 05Goal, 07OKR-Work: Provision Cassandra + Data Gateway resources for Tone Check - https://phabricator.wikimedia.org/T408129#12494731 (10Gehel) [12:29:20] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12494887 (10OKarakaya-WMF) deployment to the new namespace worked well 🎉 ` time curl -sS --max-time 900 \ "https://inference.svc... [13:07:11] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate revise-tone-task-generator GPU placement to vram-based node label - https://phabricator.wikimedia.org/T442877#12495094 (10gkyziridis) ==== Deployed ==== `lang=Shell $ kube_env revise-tone-task-generator ml-serve-eqiad $ kubectl get pods -o... [13:07:14] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate revise-tone-task-generator GPU placement to vram-based node label - https://phabricator.wikimedia.org/T442877#12495096 (10gkyziridis) 05Open→03Resolved [13:07:37] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate ml-services GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T441680#12495098 (10gkyziridis) [13:22:29] 07artificial-intelligence, 06Product Safety and Integrity, 06Security-Team, 10WMF-Claude, and 2 others: Create architectural / dependency security review skill for wmf-claude - https://phabricator.wikimedia.org/T435218#12495123 (10ASanford-WMF) p:05Triage→03Low [17:19:30] 06Machine-Learning-Team (Q2 FY2026-27): Scale TTS generation services to finish a batch run within a day - https://phabricator.wikimedia.org/T441983#12496321 (10kevinbazira) The long section that previously failed in T441983#12487788, now generates successfully in the latest 50 article batch-generation run: P970... [18:35:16] 06Machine-Learning-Team (Q2 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2026-27 Text-to-Speech V2 Experiment - https://phabricator.wikimedia.org/T441938#12496540 (10kevinbazira) **Weekly Update:** TTS v2 now generates every section of an article, not just the lead, and a 50-article batch-generation completes in ~1...