[06:47:44] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12476139 (10kevinbazira) hi @atsuko, thank you for working on this Airflow upgrade. > 1) there is even any need to actually test DA... [08:54:07] 06Machine-Learning-Team (Q2 FY2026-27): Migrate LiftWing LLMs to LLMInferenceService - https://phabricator.wikimedia.org/T441270#12476475 (10isarantopoulos) @BWojtowicz we should also update the documentation on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Large_Language_Models#KServe_Predict_En... [12:39:58] (03CR) 10Zabe: "Did you test this?" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1353433 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup-claude) [13:06:19] (03CR) 10Ladsgroup: "In the early versions, no because it was an extremely simple change. But newer versions probably should have since it's bigger. OTOH, ther" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1353433 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup-claude) [13:40:14] 06Machine-Learning-Team (Q2 FY2026-27): Integrate RevertRisk-Wikidata model with the Linked Artifacts Cache (Hoarde) - https://phabricator.wikimedia.org/T442542 (10gkyziridis) 03NEW [13:47:10] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12477868 (10atsuko) a:03atsuko [14:42:35] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Persistence: Integrate RevertRisk-Wikidata model with the Linked Artifacts Cache (Hoarde) - https://phabricator.wikimedia.org/T442542#12478267 (10isarantopoulos) [14:57:24] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12478450 (10kevinbazira) [16:06:15] (03CR) 10Zabe: [C:03+2] "Yeah, sounds good." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1353433 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup-claude) [16:08:16] (03Merged) 10jenkins-bot: Fix two bugs on PurgeScoreCache [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1353433 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup-claude) [16:30:49] FIRING: KubernetesPodUnschedulable: ... [16:30:49] Pod reference-need-predictor-00004-deployment-66df54b96-dvghn in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [16:30:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [16:38:49] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12479448 (10RKemper) a:05atsuko→03RKemper [16:39:29] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12479456 (10RKemper) [16:44:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [16:44:49] Deployment reference-need-predictor-00004-deployment in revision-models at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s-mlserve&var-namespace=revision-models&var-deployment=reference-need-predictor-00004-deployment - ... [16:44:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:49:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [16:49:49] Deployment reference-need-predictor-00004-deployment in revision-models at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s-mlserve&var-namespace=revision-models&var-deployment=reference-need-predictor-00004-deployment - ... [16:49:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:50:49] RESOLVED: KubernetesPodUnschedulable: ... [16:50:49] Pod reference-need-predictor-00004-deployment-66df54b96-dvghn in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [16:50:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [20:12:49] FIRING: KubernetesPodUnschedulable: ... [20:12:49] Pod reference-need-predictor-00004-deployment-66df54b96-fb97c in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [20:12:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [20:17:49] RESOLVED: KubernetesPodUnschedulable: ... [20:17:49] Pod reference-need-predictor-00004-deployment-66df54b96-fb97c in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [20:17:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [21:57:49] FIRING: KubernetesPodUnschedulable: ... [21:57:49] Pod reference-need-predictor-00004-deployment-66df54b96-fj4rz in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [21:57:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable [22:02:49] RESOLVED: KubernetesPodUnschedulable: ... [22:02:49] Pod reference-need-predictor-00004-deployment-66df54b96-fj4rz in revision-models at codfw cannot be scheduled - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=codfw%2Bprometheus/k8s-mlserve&var-namespace=revision-models&var-container=All - ... [22:02:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesPodUnschedulable