[05:12:46] (03PS1) 10KartikMistry: WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) [05:14:07] (03CR) 10CI reject: [V:04-1] WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) (owner: 10KartikMistry) [09:20:45] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: ml-serve-eqiad: Repartition two MI300X nodes from 24GB to 96GB partitions - https://phabricator.wikimedia.org/T436928#12353912 (10DPogorzelski-WMF) Seems that 1.31.0.9 of amd-k8s-device-plugin fixes the i... [09:24:25] 06Machine-Learning-Team (Q1 FY2026-27), 06Product Safety and Integrity, 10WikimediaAntiAbuse-AbuseReview, 05FY25-26 DE4.1 Platform capacity: AbuseReview: CoPE model URL on production is incorrect - https://phabricator.wikimedia.org/T438894#12353929 (10kevinbazira) >>! In T438894#12351817, @isarantopoulos w... [09:40:17] 06Machine-Learning-Team (Q1 FY2026-27), 06Product Safety and Integrity, 10WikimediaAntiAbuse-AbuseReview, 05FY25-26 DE4.1 Platform capacity: AbuseReview: CoPE model URL on production is incorrect - https://phabricator.wikimedia.org/T438894#12353966 (10isarantopoulos) 05Open→03Resolved [10:42:13] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 10Community Feedback (Growth), 06Growth-Team: AI/ML model update request: Named Entity Recognition for Add-a-Link - https://phabricator.wikimedia.org/T405185#12354135 (10Chipmunkdavis) @OKarakaya-WMF Special:Diff/1366779612 on en.wiki [12:14:18] (03PS2) 10KartikMistry: WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) [12:16:05] (03CR) 10CI reject: [V:04-1] WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) (owner: 10KartikMistry) [12:21:14] (03PS3) 10KartikMistry: WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) [12:22:25] (03CR) 10CI reject: [V:04-1] WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) (owner: 10KartikMistry) [12:33:31] (03PS4) 10KartikMistry: WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) [12:34:36] (03CR) 10CI reject: [V:04-1] WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) (owner: 10KartikMistry) [12:47:51] (03PS5) 10KartikMistry: WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) [13:39:12] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic: Enable For You TTS experiment via remote config flag - https://phabricator.wikimedia.org/T438845#12354985 (10WRai-WMF) https://gerrit.wikimedia.org/r/c/mediawiki/services/wikifeeds/+/1344091 [13:46:49] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: ml-serve-eqiad: Repartition two MI300X nodes from 24GB to 96GB partitions - https://phabricator.wikimedia.org/T436928#12355045 (10DPogorzelski-WMF) The new package fixes the issue of detecting the count o... [13:47:29] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: ml-serve-eqiad: Repartition two MI300X nodes from 24GB to 96GB partitions - https://phabricator.wikimedia.org/T436928#12355051 (10DPogorzelski-WMF) I will now reapply the partitioning change [15:50:39] 06Machine-Learning-Team (Q1 FY2026-27): Evaluate LibreChat and Open WebUI for LiftWing Studio Productionization - https://phabricator.wikimedia.org/T438987 (10achou) 03NEW [16:00:49] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience, 07Epic: Fix P1 issues if Text-to-Speech experiment becomes permanent - https://phabricator.wikimedia.org/T438762#12355946 (10HNordeenWMF) [16:00:57] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience, 07Epic: Fix P1 issues if Text-to-Speech experiment becomes permanent - https://phabricator.wikimedia.org/T438762#12355949 (10HNordeenWMF) p:05Triage→03Low [16:21:42] 06Machine-Learning-Team (Q1 FY2026-27): Evaluate LibreChat and Open WebUI for LiftWing Studio Productionization - https://phabricator.wikimedia.org/T438987#12356045 (10achou) Here are a few areas I think are worth comparing more closely: - MCP / agent workflows - Access control and LiteLLM integration - User-fac... [16:31:18] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]): Enable For You TTS experiment via remote config flag - https://phabricator.wikimedia.org/T438845#12356100 (10HNordeenWMF) [16:34:02] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]): TTS MVP within "For you" Data Validation - https://phabricator.wikimedia.org/T438506#12356136 (10Seddon) p:05Triage→03High [16:34:41] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 23 - Oct 06]): TTS MVP within "For you" Data Validation - https://phabricator.wikimedia.org/T438506#12356149 (10Seddon) [17:58:49] FIRING: [25x] KubernetesDeploymentUnavailableReplicas: Deployment arwiki-goodfaith-predictor-00005-deployment in revscoring-editquality-goodfaith at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [18:03:56] FIRING: [25x] KubernetesDeploymentUnavailableReplicas: Deployment arwiki-goodfaith-predictor-00005-deployment in revscoring-editquality-goodfaith at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [18:14:27] RESOLVED: [24x] KubernetesDeploymentUnavailableReplicas: Deployment arwiki-goodfaith-predictor-00005-deployment in revscoring-editquality-goodfaith at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [19:27:31] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12356979 (10GitHubPRBot) Williamrai closed https://github.com/wikimedia/apps-andro... [20:46:33] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12357179 (10HNordeenWMF) [20:47:57] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12357182 (10HNordeenWMF) @WRai-WMF, @Dbrant I updated the A/B test assignment re... [21:07:55] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12357262 (10GitHubPRBot) Williamrai opened https://github.com/wikimedia/apps-andro... [22:10:42] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12357411 (10ABorbaWMF) Tested on: 5068-alpha-2026-09-23 Pixel 8, Android 16 Galaxy...