[04:04:26] (03PS1) 10Santhosh: SECURITY: Upgrade brotli package [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1360019 (https://phabricator.wikimedia.org/T442253) [04:07:09] (03CR) 10Santhosh: [C:03+2] "Code already reviewed in the phab ticket. self merging" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1360019 (https://phabricator.wikimedia.org/T442253) (owner: 10Santhosh) [04:08:10] (03Merged) 10jenkins-bot: SECURITY: Upgrade brotli package [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1360019 (https://phabricator.wikimedia.org/T442253) (owner: 10Santhosh) [05:10:07] 06Machine-Learning-Team (Q2 FY2026-27): Scale TTS generation services to finish a batch run within a day - https://phabricator.wikimedia.org/T441983#12487788 (10kevinbazira) We ran a batch-generation job for 50 random articles, generating all sections in the article (not only the lead section as v1) using the cu... [08:14:28] (03PS1) 10Bartosz Wójtowicz: revscoring: upgrade certifi to 2026.10.23 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360337 [08:22:29] (03CR) 10AikoChou: [C:03+1] revscoring: upgrade certifi to 2026.10.23 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360337 (owner: 10Bartosz Wójtowicz) [08:22:56] (03CR) 10Bartosz Wójtowicz: [C:03+2] revscoring: upgrade certifi to 2026.10.23 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360337 (owner: 10Bartosz Wójtowicz) [08:24:13] (03Merged) 10jenkins-bot: revscoring: upgrade certifi to 2026.10.23 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360337 (owner: 10Bartosz Wójtowicz) [08:55:26] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Kubernetes, 13Patch-For-Review: Extend KServe's LLMInferenceService by Gateway API Inference Extension and KEDA autoscaling - https://phabricator.wikimedia.org/T436650#12488210 (10DPogorzelski-WMF) 05Open→03In progr... [10:39:41] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Add 'test/unit//' paths to Zuul trigger file filters - https://phabricator.wikimedia.org/T440288#12488594 (10gkyziridis) [10:47:37] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Add 'test/unit//' paths to Zuul trigger file filters - https://phabricator.wikimedia.org/T440288#12488627 (10gkyziridis) All the models mentioned in the description are configured correctly in the config repository except the two "... [10:48:00] 10Lift-Wing, 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Add 'test/unit//' paths to Zuul trigger file filters - https://phabricator.wikimedia.org/T440288#12488628 (10gkyziridis) 05Open→03Resolved a:03gkyziridis [11:16:43] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate ml-services GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T441680#12488763 (10gkyziridis) 05Open→03In progress a:03gkyziridis [11:17:27] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate ml-services GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T441680#12488767 (10gkyziridis) [11:18:18] (03PS1) 10Kevin Bazira: tts-section-generator: split long sections across multiple requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360378 (https://phabricator.wikimedia.org/T441983) [11:18:36] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12488773 (10gkyziridis) 05Open→03In progress [11:19:32] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Define and implement SLOs for the public LLM inference services - https://phabricator.wikimedia.org/T440307#12488774 (10gkyziridis) 05Open→03In progress a:03gkyziridis [11:33:33] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12488834 (10achou) Hey @OKarakaya-WMF, could you help me understand a bit more about why scale-to-zero isn't working with the current... [11:42:32] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12488871 (10OKarakaya-WMF) Hey @achou , thank you for the comments. We are discussing the issue and the fix in this thread with @klau... [11:53:02] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate ml-services GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T441680#12488900 (10achou) Hey @gkyziridis, just a quick heads up: if you're planning to work on a model, please create a subtask for it so others can pick... [12:00:24] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Create a namespace on ml-serve-eqiad for batch LLM workloads (llm-batch) - https://phabricator.wikimedia.org/T442150#12488937 (10isarantopoulos) I know I wrote this patch, but it has been more than 3 years so I don't have any recollection of it other... [12:15:13] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate ml-services GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T441680#12489228 (10gkyziridis) [12:29:21] 06Machine-Learning-Team (Q2 FY2026-27): Migrate edit-check GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T442856 (10gkyziridis) 03NEW [12:29:38] 06Machine-Learning-Team (Q2 FY2026-27): Migrate edit-check GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T442856#12489287 (10gkyziridis) 05Open→03Resolved [12:30:29] 06Machine-Learning-Team (Q2 FY2026-27): Migrate revertrisk-wikidata GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T442857 (10gkyziridis) 03NEW [12:31:00] 06Machine-Learning-Team (Q2 FY2026-27): Migrate revertrisk-wikidata GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T442857#12489305 (10gkyziridis) 05Open→03In progress [13:27:21] 06Machine-Learning-Team (Q2 FY2026-27), 06Discovery-Search (2026.09.01 - 2026.10.02), 13Patch-For-Review: Create SLO for the jina-embeddings model server using Sloth - https://phabricator.wikimedia.org/T439245#12489539 (10OKarakaya-WMF) Reviewed by @dcausse . I'll move the SLO page from draft to approved if... [13:27:40] 06Machine-Learning-Team (Q2 FY2026-27), 06Discovery-Search (2026.09.01 - 2026.10.02), 13Patch-For-Review: Create SLO for the jina-embeddings model server using Sloth - https://phabricator.wikimedia.org/T439245#12489541 (10OKarakaya-WMF) [13:32:22] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity, 13Patch-For-Review: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12489566 (10OKarakaya-WMF) Hello @RLazarus , Would you mind reviewing the slo page for CoPE-B_Model. https://wikitech.w... [13:32:57] 06Machine-Learning-Team (Q2 FY2026-27), 06Discovery-Search (2026.09.01 - 2026.10.02), 13Patch-For-Review: Create SLO for the jina-embeddings model server using Sloth - https://phabricator.wikimedia.org/T439245#12489572 (10OKarakaya-WMF) Hello @RLazarus , Would you mind reviewing the slo page for JinaEmbeddi... [13:35:03] (03CR) 10Bartosz Wójtowicz: "Thank you for working on this! Leaving a couple of questions :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360378 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [13:36:49] FIRING: [2x] KubernetesPodUnschedulable: ... [13:36:49] Pod revertrisk-wikidata-predictor-00002-deployment-54cfc798f4-s7qdv in revertrisk at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revertrisk&var-container=All - https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [13:51:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [13:51:49] Deployment revertrisk-wikidata-predictor-00002-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00002-deployment - ... [13:51:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [13:56:49] FIRING: [2x] KubernetesPodUnschedulable: ... [13:56:49] Pod revertrisk-wikidata-predictor-00002-deployment-54cfc798f4-s7qdv in revertrisk at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revertrisk&var-container=All - https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [14:01:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [14:01:49] Deployment revertrisk-wikidata-predictor-00002-deployment in revertrisk at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-wikidata-predictor-00002-deployment - ... [14:01:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [14:01:51] RESOLVED: [2x] KubernetesPodUnschedulable: ... [14:01:57] Pod revertrisk-wikidata-predictor-00002-deployment-54cfc798f4-s7qdv in revertrisk at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revertrisk&var-container=All - https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [14:06:51] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate revertrisk-wikidata GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T442857#12489750 (10gkyziridis) ==== Deployed ==== `lang=Shell $ kube_env revertrisk ml-serve-eqiad $ kubectl get pods -o wide | grep wikidata... [14:07:00] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate revertrisk-wikidata GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T442857#12489751 (10gkyziridis) 05In progress→03Resolved [14:08:05] 06Machine-Learning-Team (Q2 FY2026-27), 13Patch-For-Review: Migrate ml-services GPU placement to vram-based node labels - https://phabricator.wikimedia.org/T441680#12489754 (10gkyziridis) [14:10:12] 06Machine-Learning-Team (Q2 FY2026-27): Allow deploy users to delete Knative revisions - https://phabricator.wikimedia.org/T442872 (10BWojtowicz-WMF) 03NEW [14:14:01] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE: Allow deploy users to delete Knative revisions - https://phabricator.wikimedia.org/T442872#12489788 (10isarantopoulos) [14:18:06] 06Machine-Learning-Team (Q2 FY2026-27): Migrate revise-tone-task-generator GPU placement to vram-based node label - https://phabricator.wikimedia.org/T442877 (10gkyziridis) 03NEW [14:39:22] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work, 13Patch-For-Review: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12490027 (10klausman) Destroyed all helmfile releases for article-descriptions: ` root@de... [15:19:46] (03PS2) 10Kevin Bazira: tts-section-generator: split long sections across multiple requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360378 (https://phabricator.wikimedia.org/T441983) [15:19:55] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work, 13Patch-For-Review: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12490424 (10klausman) [15:20:13] (03CR) 10Kevin Bazira: tts-section-generator: split long sections across multiple requests (036 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1360378 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [15:46:31] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Kubernetes: Extend KServe's LLMInferenceService by Gateway API Inference Extension and KEDA autoscaling - https://phabricator.wikimedia.org/T436650#12490613 (10DPogorzelski-WMF) Progress on ml-staging-codfw (ml-serve eqi... [15:59:43] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work, 13Patch-For-Review: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12490746 (10klausman) [16:46:14] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12491013 (10isarantopoulos) a:05klausman→03None [17:58:16] 06Machine-Learning-Team: SLOBudgetBurn - https://phabricator.wikimedia.org/T430669#12491455 (10phaultfinder)