[05:59:49] FIRING: KubernetesPodUnschedulable: ... [05:59:49] Pod reference-need-predictor-00004-deployment-66df54b96-7bj2j in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [05:59:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [06:04:49] RESOLVED: KubernetesPodUnschedulable: ... [06:04:49] Pod reference-need-predictor-00004-deployment-66df54b96-7bj2j in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [06:04:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [06:18:26] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12482889 (10kevinbazira) Following T435481#12476139, we added [example_training_dag](https://airflow-ml.wikimedia.org/dags/example_t... [06:30:49] FIRING: KubernetesPodUnschedulable: ... [06:30:49] Pod reference-need-predictor-00004-deployment-66df54b96-5vrq8 in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [06:30:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [06:35:49] RESOLVED: KubernetesPodUnschedulable: ... [06:35:49] Pod reference-need-predictor-00004-deployment-66df54b96-5vrq8 in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [06:35:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [07:10:34] (03PS1) 10Kevin Bazira: tts-generator: fix parallel article generation failures [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359737 (https://phabricator.wikimedia.org/T441983) [07:44:40] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12482973 (10kevinbazira) [08:00:35] (03CR) 10Bartosz Wójtowicz: "Thank you for the fix! Left 1 small comment" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359737 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [08:41:37] (03PS2) 10Kevin Bazira: tts-generator: fix parallel article generation failures [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359737 (https://phabricator.wikimedia.org/T441983) [08:42:18] (03CR) 10Kevin Bazira: tts-generator: fix parallel article generation failures (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359737 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [08:43:29] (03CR) 10Bartosz Wójtowicz: [C:03+1] tts-generator: fix parallel article generation failures (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359737 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [08:44:22] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12483078 (10klausman) File an SMC ticket with the ID CS-00... [08:44:40] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12483080 (10klausman) 05In progress→03Stalled p:05Tri... [08:46:40] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Decide on the future shape of ML/LW SLOs, and how they are monitored - https://phabricator.wikimedia.org/T434525#12483096 (10klausman) [08:46:44] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity, 13Patch-For-Review: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12483097 (10klausman) [08:46:47] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Define and implement SLOs for the public LLM inference services - https://phabricator.wikimedia.org/T440307#12483098 (10klausman) [08:46:52] 06Machine-Learning-Team (Q2 FY2026-27), 06Discovery-Search (2026.09.01 - 2026.10.02), 13Patch-For-Review: Create SLO for the jina-embeddings model server using Sloth - https://phabricator.wikimedia.org/T439245#12483099 (10klausman) [08:51:53] (03CR) 10Kevin Bazira: [C:03+2] tts-generator: fix parallel article generation failures [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359737 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [08:59:57] (03Merged) 10jenkins-bot: tts-generator: fix parallel article generation failures [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359737 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [09:01:28] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Upgrade vLLM in AMD ROCm base image to v0.32.0 - https://phabricator.wikimedia.org/T442138#12483128 (10BWojtowicz-WMF) I built a new `amd-vllm032` image on ml-lab1002 and verified that it works on our Qwen3-14B. Versions: vLLM 0.32.0,... [09:29:30] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity, 13Patch-For-Review: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12483233 (10OKarakaya-WMF) [09:29:47] 06Machine-Learning-Team (Q2 FY2026-27), 06Discovery-Search (2026.09.01 - 2026.10.02), 13Patch-For-Review: Create SLO for the jina-embeddings model server using Sloth - https://phabricator.wikimedia.org/T439245#12483234 (10OKarakaya-WMF) [09:34:32] (03PS1) 10Ladsgroup: Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359767 (https://phabricator.wikimedia.org/T440358) [09:34:44] (03PS1) 10Ladsgroup: Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359768 (https://phabricator.wikimedia.org/T440358) [09:34:56] (03CR) 10Ladsgroup: [C:03+2] Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359767 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup) [09:34:59] (03CR) 10Ladsgroup: [C:03+2] Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359768 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup) [09:36:47] (03Merged) 10jenkins-bot: Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359767 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup) [09:37:39] (03Merged) 10jenkins-bot: Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359768 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup) [10:21:21] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity, 13Patch-For-Review: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12483426 (10kostajh) The dashboard looks good to me, thanks! [10:26:15] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity, 13Patch-For-Review: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12483463 (10OKarakaya-WMF) [10:27:31] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity, 13Patch-For-Review: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12483478 (10OKarakaya-WMF) thank you @kostajh , I'm closing the task. We will keep an eye on the usage. [10:28:57] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity, 13Patch-For-Review: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12483494 (10OKarakaya-WMF) 05Open→03Resolved [10:34:39] 06Machine-Learning-Team (Q2 FY2026-27): Scale TTS generation services to finish a batch run within a day - https://phabricator.wikimedia.org/T441983#12483526 (10kevinbazira) The TTS generation service now processes articles in parallel without failures. The two things that caused those failures have now been fi... [10:34:49] 06Machine-Learning-Team (Q2 FY2026-27): Scale TTS generation services to finish a batch run within a day - https://phabricator.wikimedia.org/T441983#12483537 (10kevinbazira) [11:03:30] 06Machine-Learning-Team (Q2 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2026-27 Text-to-Speech V2 Experiment - https://phabricator.wikimedia.org/T441938#12483645 (10kevinbazira) [11:15:55] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12483734 (10OKarakaya-WMF) Sharing an alternative solution that we may offer in the future: https://docs.vllm.ai/en/stable/cli/run-ba... [11:21:00] 06Machine-Learning-Team (Q2 FY2026-27): Scale TTS generation services to finish a batch run within a day - https://phabricator.wikimedia.org/T441983#12483748 (10kevinbazira) [11:32:43] 06Machine-Learning-Team (Q2 FY2026-27): Automate recurring TTS batch generation with a daily Airflow DAG - https://phabricator.wikimedia.org/T441987#12483806 (10kevinbazira) [11:38:27] 06Machine-Learning-Team (Q2 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2026-27 Text-to-Speech V2 Experiment - https://phabricator.wikimedia.org/T441938#12483857 (10kevinbazira) [12:11:32] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12483916 (10atsuko) >>! In T435481#12482889, @kevinbazira wrote: > Following T435481#12476139, we added [example_training_dag](https... [14:32:55] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12484718 (10DPogorzelski-WMF) 05Open→03Resolved a:03DPogorzelski-WMF [14:33:22] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: monitoring: Grafana dashboard for LLM serving on MI300X - https://phabricator.wikimedia.org/T429237#12484725 (10DPogorzelski-WMF) 05Open→03Resolved a:03DPogorzelski-WMF [14:36:27] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12484771 (10DPogorzelski-WMF) 05Open→03Resolved I will mark this as solved since we have a solution.... [14:36:48] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 05Goal: Setup MI300X nodes for LLM serving - https://phabricator.wikimedia.org/T424322#12484782 (10DPogorzelski-WMF) 05Open→03Resolved [15:22:19] 06Machine-Learning-Team, 05MW-1.47-notes (1.47.0-wmf.24; 2026-10-13), 05MW-1.48-notes (1.48.0-wmf.1; 2026-10-20): Lots of old data in ores_classification table - https://phabricator.wikimedia.org/T440358#12485005 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup Slight better now: ` mysql:research@dbst... [15:29:42] (03PS1) 10Gkyziridis: revertrisk_wikidata: Integrate LAC (on-deamdn) flow [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1359881 (https://phabricator.wikimedia.org/T433699) [15:36:53] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12485067 (10gkyziridis) ==== On-demand flow: implemented and tested locally (Lift Wing side) ==== Following the on-demand flow diagram... [16:10:44] 06Machine-Learning-Team (Q2 FY2026-27): Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12485268 (10OKarakaya-WMF) the pods are created after setting the credentials. the pods are health but the requests fail As I understand, we will need to... [16:28:19] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12485444 (10OKarakaya-WMF) yes it worked with minreplica:1 Now let's set it back to 0 and see what happens. ` curl "https://inference... [16:50:10] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12485592 (10OKarakaya-WMF) I confirm the issue is the minreplica. I'll create a new patch to fix it in knative and it impact all servi... [19:05:06] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12486242 (10Ottomata) > If LAC fails (error, timeout, connection refused), the isvc logs a warning and predicts locally, so users still... [22:16:24] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 10Community Feedback (Growth), 06Growth-Team: Scaling of case sensitivity for Link Suggestions - https://phabricator.wikimedia.org/T436545#12487077 (10Quiddity) Noting: A fresh bug-report from Italian Wikipedia [[ https://www.mediawiki.org/w/index.php?ti...