[00:41:24] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12357715 (10kevinbazira) The second batch generation (backfill) completed. Coverage is now 7,763 of 7,800 articles (99.5%), up from 7,716. Based on the backfill report,... [01:31:36] 06Machine-Learning-Team (Q1 FY2026-27): Over-limit text is misreported as a failure and never split - https://phabricator.wikimedia.org/T438647#12357787 (10kevinbazira) The backfill completed as shown in T436758#12357715. Over-limit text is now recorded as a skip rather than retried and marked failed. [01:39:01] 06Machine-Learning-Team (Q1 FY2026-27): Over-limit text is misreported as a failure and never split - https://phabricator.wikimedia.org/T438647#12357790 (10kevinbazira) [07:16:13] (03PS1) 10Kevin Bazira: tts-section-generator: strip scripts espeak cannot read before synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344536 (https://phabricator.wikimedia.org/T438647) [07:31:53] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: LiteLLM: Test locally as a gateway for Lift Wing models - https://phabricator.wikimedia.org/T438691#12358376 (10gkyziridis) [07:53:59] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Update kserve helm chart to include LLMInferenceService - https://phabricator.wikimedia.org/T433973#12358460 (10Gehel) [07:54:37] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 05Goal: Setup MI300X nodes for LLM serving - https://phabricator.wikimedia.org/T424322#12358463 (10Gehel) [08:05:03] (03CR) 10Ozge: [C:03+1] tts-section-generator: strip scripts espeak cannot read before synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344536 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [08:06:51] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Update kserve helm chart to include LLMInferenceService - https://phabricator.wikimedia.org/T433973#12358479 (10Gehel) We're waiting on #machine-learning-team to confirm that we can move forward. [08:07:11] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Update kserve helm chart to include LLMInferenceService - https://phabricator.wikimedia.org/T433973#12358481 (10Gehel) a:05DPogorzelski-WMF→03isarantopoulos [08:07:48] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: strip scripts espeak cannot read before synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344536 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [08:14:29] (03Merged) 10jenkins-bot: tts-section-generator: strip scripts espeak cannot read before synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344536 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [10:36:52] (03PS1) 10Kevin Bazira: tts-section-generator: keep symbols by default when stripping unreadable scripts [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344624 (https://phabricator.wikimedia.org/T438647) [10:50:32] 06Machine-Learning-Team (Q1 FY2026-27): Evaluate LibreChat and Open WebUI for LiftWing Studio Productionization - https://phabricator.wikimedia.org/T438987#12359394 (10OKarakaya-WMF) I've compared openwebui and librechat on authorization via litellm. {F104579981, width=700} Main idea is to match litellm teams... [10:51:15] (03CR) 10Ozge: [C:03+1] tts-section-generator: keep symbols by default when stripping unreadable scripts [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344624 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [10:51:42] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: keep symbols by default when stripping unreadable scripts [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344624 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [11:00:02] (03Merged) 10jenkins-bot: tts-section-generator: keep symbols by default when stripping unreadable scripts [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344624 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [11:06:14] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: ml-serve-eqiad: Repartition two MI300X nodes from 24GB to 96GB partitions - https://phabricator.wikimedia.org/T436928#12359461 (10DPogorzelski-WMF) 05Open→03Resolved ml-serve1013 and ml-serve1014... [12:30:20] (03PS1) 10Kevin Bazira: tts-section-generator: strip script symbols by block not by name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344654 (https://phabricator.wikimedia.org/T438647) [12:36:21] (03CR) 10Ozge: [C:03+1] tts-section-generator: strip script symbols by block not by name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344654 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [12:38:38] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: strip script symbols by block not by name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344654 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [12:42:13] (03Merged) 10jenkins-bot: tts-section-generator: strip script symbols by block not by name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344654 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [13:39:27] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12360126 (10Gehel) [13:39:35] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12360125 (10Gehel) [13:58:16] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12360234 (10GitHubPRBot) Williamrai opened https://github.com/wikimedia/apps-andro... [14:04:50] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12360284 (10DPogorzelski-WMF) 05In progress→03Resolved [14:06:04] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: LLM observability - https://phabricator.wikimedia.org/T429597#12360293 (10DPogorzelski-WMF) 05Open→03Resolved [14:54:28] (03PS1) 10Sbisson: reference-risk: accept a single domain as input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) [17:43:44] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12361293 (10GitHubPRBot) voyagerfan closed https://github.com/wikimedia/apps-andro... [20:06:52] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12361774 (10GitHubPRBot) cooltey closed https://github.com/wikimedia/apps-android-... [21:53:30] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12362080 (10ABorbaWMF) This is working as expected. Misunderstood the requirement...