[06:32:00] (03PS1) 10Kevin Bazira: tts-section-generator: fold latin characters espeak cannot pronounce before synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1343849 (https://phabricator.wikimedia.org/T438647) [07:33:38] (03CR) 10Ozge: [C:03+1] tts-section-generator: fold latin characters espeak cannot pronounce before synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1343849 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [07:50:50] (03CR) 10Kevin Bazira: [C:03+2] tts-section-generator: fold latin characters espeak cannot pronounce before synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1343849 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [07:58:47] (03Merged) 10jenkins-bot: tts-section-generator: fold latin characters espeak cannot pronounce before synthesis [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1343849 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [08:16:05] 06Machine-Learning-Team (Q1 FY2026-27): liftwing-studio: Set up a shared local LibreChat environment - https://phabricator.wikimedia.org/T438673#12348727 (10achou) Some possible next steps from today's standup: * **LiteLLM integration**: explore how LibreChat and LiteLLM could work together, including where MCP... [08:32:44] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: LiteLLM: Test locally as a gateway for Lift Wing models - https://phabricator.wikimedia.org/T438691#12348801 (10isarantopoulos) The following third party providers have been added as examples: openrouter, mistral [09:40:12] 06Machine-Learning-Team (Q1 FY2026-27): liftwing-studio: Set up a shared local LibreChat environment - https://phabricator.wikimedia.org/T438673#12349143 (10OKarakaya-WMF) [13:14:33] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic: Enable For You TTS experiment via remote config flag - https://phabricator.wikimedia.org/T438845 (10Seddon) 03NEW [13:14:47] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic: Enable For You TTS experiment via remote config flag - https://phabricator.wikimedia.org/T438845#12350048 (10Seddon) p:05Triage→03High [13:15:32] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: liftwing-studio: Set up a shared local LibreChat environment - https://phabricator.wikimedia.org/T438673#12350052 (10OKarakaya-WMF) https://gitlab.wikimedia.org/repos/machine-learning/liftwing-studio/-/merge_requests/11 - Librechat can use litellm a... [13:42:17] 06Machine-Learning-Team, 10Ceph, 06Data-Platform-SRE, 06Infrastructure-Foundations, and 3 others: Design and build the next generation of container-registry service for the WMF production realm - https://phabricator.wikimedia.org/T413080#12350158 (10elukey) Since the last update a lot of things changed, go... [14:10:17] 06Machine-Learning-Team, 10Ceph, 06Data-Platform-SRE, 06Infrastructure-Foundations, and 3 others: Design and build the next generation of container-registry service for the WMF production realm - https://phabricator.wikimedia.org/T413080#12350284 (10MatthewVernon) >>! In T413080#12350158, @elukey wrote: >... [14:21:35] 06Machine-Learning-Team, 10Ceph, 06Data-Platform-SRE, 06Infrastructure-Foundations, and 3 others: Design and build the next generation of container-registry service for the WMF production realm - https://phabricator.wikimedia.org/T413080#12350338 (10elukey) >>! In T413080#12350284, @MatthewVernon wrote: >>... [14:50:49] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment cope-a-9b-predictor-00004-deployment in experimental at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [14:51:44] FIRING: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=llm&var-backend=knative-serving.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [15:23:36] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: liftwing-studio: Set up a shared local LibreChat environment - https://phabricator.wikimedia.org/T438673#12350880 (10achou) I spent some time looking into LiteLLM's [[ https://docs.litellm.ai/docs/a2a | Agent and MCP gateways ]] today, mainly to under... [15:48:39] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: ml-serve-eqiad: Repartition two MI300X nodes from 24GB to 96GB partitions - https://phabricator.wikimedia.org/T436928#12350985 (10DPogorzelski-WMF) I had to rollback the change as I ran into issues with t... [15:50:49] RESOLVED: [2x] KubernetesDeploymentUnavailableReplicas: Deployment cope-a-9b-predictor-00004-deployment in experimental at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [15:51:44] RESOLVED: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=llm&var-backend=knative-serving.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [15:52:18] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Add a link at English Wikipedia: do not suggest a link next to an existing one that would appear to be a single link - https://phabricator.wikimedia.org/T437929#12351009 (10Trizek-WMF) >>! In T437929#12338900, @Urbanecm_WMF wrote: > @Trizek-... [16:41:46] 06Machine-Learning-Team, 06Editing-team, 06Growth-Team, 10Revise-Tone-Structured-Task, and 2 others: Communications: Revise Tone Structured Task release communication for: eswiki, ruwiki, jawiki, nlwiki, zhwiki, dewiki - https://phabricator.wikimedia.org/T435836#12351374 (10KStoller-WMF) [17:50:48] 06Machine-Learning-Team (Q1 FY2026-27), 06Product Safety and Integrity, 10WikimediaAntiAbuse-AbuseReview, 05FY25-26 DE4.1 Platform capacity: AbuseReview: CoPE model URL on production is incorrect - https://phabricator.wikimedia.org/T438894#12351764 (10isarantopoulos) [17:55:51] 06Machine-Learning-Team (Q1 FY2026-27), 06Product Safety and Integrity, 10WikimediaAntiAbuse-AbuseReview, 05FY25-26 DE4.1 Platform capacity: AbuseReview: CoPE model URL on production is incorrect - https://phabricator.wikimedia.org/T438894#12351803 (10isarantopoulos) - these services exist only in eqiad si... [17:57:01] 06Machine-Learning-Team (Q1 FY2026-27), 06Product Safety and Integrity, 10WikimediaAntiAbuse-AbuseReview, 05FY25-26 DE4.1 Platform capacity: AbuseReview: CoPE model URL on production is incorrect - https://phabricator.wikimedia.org/T438894#12351812 (10Dreamy_Jazz) With the URL fix deployed, the logs have s... [17:58:05] 06Machine-Learning-Team (Q1 FY2026-27), 06Product Safety and Integrity, 10WikimediaAntiAbuse-AbuseReview, 05FY25-26 DE4.1 Platform capacity: AbuseReview: CoPE model URL on production is incorrect - https://phabricator.wikimedia.org/T438894#12351817 (10isarantopoulos) A follow up item before we close this t... [20:04:28] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team: Add a link at English Wikipedia: do not suggest a link next to an existing one that would appear to be a single link - https://phabricator.wikimedia.org/T437929#12352415 (10OKarakaya-WMF) @Urbanecm_WMF , I agree it fits better in the api as... [22:49:43] FIRING: LiftWingServiceErrorRate: ... [22:49:43] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=codfw%20prometheus/k8s-mlserve&var-namespace=tts-section-generator&var-backend=tts-section-generator.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [23:09:43] RESOLVED: LiftWingServiceErrorRate: ... [23:09:43] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=codfw%20prometheus/k8s-mlserve&var-namespace=tts-section-generator&var-backend=tts-section-generator.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate