[05:38:13] (03PS1) 10Kevin Bazira: tts-section-generator: parallelize section enumeration [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1348199 (https://phabricator.wikimedia.org/T436758) [06:50:21] (03CR) 10Ozge: [C:03+1] tts-section-generator: parallelize section enumeration [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1348199 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [06:54:14] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: parallelize section enumeration [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1348199 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [07:02:56] (03Merged) 10jenkins-bot: tts-section-generator: parallelize section enumeration [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1348199 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [07:53:30] 06Machine-Learning-Team (Q1 FY2026-27): Evaluate LibreChat and Open WebUI for LiftWing Studio Productionization - https://phabricator.wikimedia.org/T438987#12388277 (10achou) 05Open→03Resolved [09:57:19] (03CR) 10AikoChou: [C:03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338218 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [10:16:09] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Add Qwen3-14B-FP8 LLMInferenceService in LiftWing staging - https://phabricator.wikimedia.org/T439659#12388629 (10DPogorzelski-WMF) @kevinbazira this is fixed now [11:18:54] 06Machine-Learning-Team (Q1 FY2026-27): Create dataset of article titles for TTS V1 - https://phabricator.wikimedia.org/T432128#12388855 (10isarantopoulos) 05Open→03Resolved Since were generating the audio files I'm marking this task as resolved [11:19:41] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12388859 (10achou) >>! In T433699#12304464, @Ottomata wrote: >> Wikimedia Enterprise — switch to the LAC endpoint > > Q: I wonder if a... [11:20:11] 06Machine-Learning-Team (Q1 FY2026-27): Add Qwen3-14B-FP8 LLMInferenceService in LiftWing staging - https://phabricator.wikimedia.org/T439659#12388860 (10kevinbazira) >>! In T439659#12388629, @DPogorzelski-WMF wrote: > @kevinbazira this is fixed now Thanks for the fix @DPogorzelski-WMF. I am now able to access... [11:45:44] 06Machine-Learning-Team (Q1 FY2026-27), 06Traffic: CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12388907 (10isarantopoulos) @ssingh o/ Just a heads up that @kevinbazira has started generating the batch of audio files that will be used in the TTS experiment that will st... [11:54:11] 06Machine-Learning-Team (Q1 FY2026-27), 06Traffic: CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12388936 (10isarantopoulos) The total size is much smaller than the initial request since we are only generating audio for the Lead section of each article. @kevinbazira can... [11:59:31] 06Machine-Learning-Team, 06Data-Platform-SRE: pytorch image builds are failing - https://phabricator.wikimedia.org/T437427#12388952 (10klausman) 05Open→03Resolved As far as I can tell, these are building fine on the weekly runs: ` build2004 $ docker images |grep amd-pytorch docker-registry.discovery.... [12:18:28] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Move llm isvcs to new partitions - https://phabricator.wikimedia.org/T439555#12389022 (10gkyziridis) ==== Qwen3-14b Alert ==== Alert: Oct 1, 2026, 7:22 PM description = The rate of errors (non-2/3/4xx return codes) in the LiftWing service llm-qwen3-14b-pre... [13:17:51] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): qwen3-14b model server: Return specific http status (400) when the prompt exceeds MAX_MODEL_LEN - https://phabricator.wikimedia.org/T439967 (10gkyziridis) 03NEW [13:18:12] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): qwen3-14b model server: Return specific http status (400) when the prompt exceeds MAX_MODEL_LEN - https://phabricator.wikimedia.org/T439967#12389206 (10gkyziridis) [13:18:26] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): qwen3-14b model server: Return specific http status (400) when the prompt exceeds MAX_MODEL_LEN - https://phabricator.wikimedia.org/T439967#12389207 (10gkyziridis) [13:37:46] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: LiteLLM: Test locally as a gateway for Lift Wing models - https://phabricator.wikimedia.org/T438691#12389342 (10isarantopoulos) [13:37:48] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: LiteLLM: Test locally as a gateway for Lift Wing models - https://phabricator.wikimedia.org/T438691#12389343 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos [14:39:22] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12389642 (10Ottomata) Hm. I was imagining just hiding LAC behind whatever endpoint you expect LiftWing HTTP callers to call. Where poss... [15:19:34] 06Machine-Learning-Team (Q1 FY2026-27): Measurements like "56.7 m" are read aloud as the letter "M" - https://phabricator.wikimedia.org/T440001 (10kevinbazira) 03NEW [15:27:34] (03PS1) 10Kevin Bazira: tts-section-generator: speak "m" and "mm" as meters and millimeters [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1348768 (https://phabricator.wikimedia.org/T440001) [15:39:19] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: speak "m" and "mm" as meters and millimeters [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1348768 (https://phabricator.wikimedia.org/T440001) (owner: 10Kevin Bazira) [15:48:42] (03Merged) 10jenkins-bot: tts-section-generator: speak "m" and "mm" as meters and millimeters [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1348768 (https://phabricator.wikimedia.org/T440001) (owner: 10Kevin Bazira) [17:06:45] 06Machine-Learning-Team (Q1 FY2026-27): Measurements like "56.7 m" are read aloud as the letter "M" - https://phabricator.wikimedia.org/T440001#12390631 (10kevinbazira) Measurements written with a bare "m" or "mm" are now read as "meters" and "millimeters" in v1 of the TTS service as shown below: `python >>> fro... [17:07:55] 06Machine-Learning-Team (Q1 FY2026-27): Measurements like "56.7 m" are read aloud as the letter "M" - https://phabricator.wikimedia.org/T440001#12390640 (10kevinbazira) [17:08:25] 06Machine-Learning-Team (Q1 FY2026-27): Measurements like "56.7 m" are read aloud as the letter "M" - https://phabricator.wikimedia.org/T440001#12390642 (10kevinbazira) 05Open→03Resolved [17:51:59] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12390769 (10kevinbazira) As the Apps team targets to [release into production](https://wikimedia.slack.com/archives/C0B016PMCCA/p1790708269042709?thread_ts=1789056139.02... [18:31:33] 06Machine-Learning-Team (Q1 FY2026-27), 06Traffic: CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12390833 (10ssingh) Some updates: After @isarantopoulos' message today and given the backends are active, I checked the cache-control stuff to make sure it would align what... [21:36:32] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 23 - Oct 06]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12391659 (10HNordeenWMF) [22:56:30] 06Machine-Learning-Team (Q1 FY2026-27): Create dataset of article titles for TTS V1 - https://phabricator.wikimedia.org/T432128#12391814 (10HNordeenWMF) Yay, thank you for all the hard work on this @isarantopoulos and @kevinbazira ! Confirming that we don't need to make an update to the CSV since the Beta ge...