[05:26:33] (03PS6) 10KartikMistry: WIP: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) [05:27:09] (03PS7) 10KartikMistry: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) [07:50:54] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12404824 (10klausman) We will be running the basleine test... [07:52:46] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Decide on the future shape of ML/LW SLOs, and how they are monitored - https://phabricator.wikimedia.org/T434525#12404826 (10klausman) SLOs arr managed in Sloth these days ([[ https://wikitech.wikimedia.org/wiki/Sloth | Wikitech page ]], [[ https://g... [08:07:23] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Decide on the future shape of ML/LW SLOs, and how they are monitored - https://phabricator.wikimedia.org/T434525#12404867 (10Gehel) p:05Triage→03Medium [08:16:17] 06Machine-Learning-Team (Q1 FY2026-27), 06Growth-Team, 10Revise-Tone-Structured-Task, 13Patch-For-Review, 07Wikimedia-production-error: No tone suggestions found from API for title {title} and revision {revision} - https://phabricator.wikimedia.org/T440218#12404890 (10OKarakaya-WMF) I've randomly checked... [08:29:25] 06Machine-Learning-Team (Q2 FY2026-27): Deploy an open-weight decision model for classification use cases - https://phabricator.wikimedia.org/T440223#12404946 (10isarantopoulos) Thanks for creating this task Fabian. We've been looking into decision models as well and there is indeed a lot of benefit in using the... [08:35:10] 06Machine-Learning-Team (Q2 FY2026-27): Deploy an open-weight decision model for classification use cases - https://phabricator.wikimedia.org/T440223#12404987 (10isarantopoulos) a:03kevinbazira [09:01:16] (03PS1) 10Kevin Bazira: decision-models: add clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) [09:24:16] 06Machine-Learning-Team (Q1 FY2026-27), 06Growth-Team, 10Revise-Tone-Structured-Task, 13Patch-For-Review, 07Wikimedia-production-error: No tone suggestions found from API for title {title} and revision {revision} - https://phabricator.wikimedia.org/T440218#12405202 (10OKarakaya-WMF) I've checked the foll... [10:24:30] 06Machine-Learning-Team (Q2 FY2026-27): tool-server: Show Wikipedia semantic search results as cards in chat (MCP Apps) - https://phabricator.wikimedia.org/T440444 (10achou) 03NEW [10:29:37] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: Qwen3.6-27B is slow - https://phabricator.wikimedia.org/T440266#12405458 (10OKarakaya-WMF) Deploying the changes: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1352020 I get the error below although we set the mi... [10:32:36] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Qwen3.6-27B is slow - https://phabricator.wikimedia.org/T440266#12405473 (10OKarakaya-WMF) I've manually updated the min max replicas and synced worked now. However, it didn't reduce the pod count to 1. Looking further. ` kubectl --kubeconfig=/etc/kub... [11:44:23] (03PS1) 10Kevin Bazira: decision-models: containerize clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352492 (https://phabricator.wikimedia.org/T440223) [12:01:49] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Qwen3.6-27B is slow - https://phabricator.wikimedia.org/T440266#12405771 (10OKarakaya-WMF) Deployment worked after manually deleting the llm-qwen36-27b service. ` kubectl delete isvc llm-qwen36-27b ` After this the configuration with single replica ap... [12:06:12] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Qwen3.6-27B is slow - https://phabricator.wikimedia.org/T440266#12405777 (10OKarakaya-WMF) I'm closing this ticket as we have: - identified the root cause of slowness as it's the load. - created an internal deployment for experiments. - did a clean up... [12:07:23] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Qwen3.6-27B is slow - https://phabricator.wikimedia.org/T440266#12405779 (10OKarakaya-WMF) 05Open→03Resolved [12:22:30] 06Machine-Learning-Team (Q2 FY2026-27): Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12405818 (10isarantopoulos) [12:24:29] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12405820 (10isarantopoulos) [12:31:24] 06Machine-Learning-Team (Q2 FY2026-27), 06Product Safety and Integrity: Define and implement SLOs for the CoPE-B model - https://phabricator.wikimedia.org/T440311#12405835 (10isarantopoulos) [13:27:43] FIRING: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=llm&var-backend=llm-qwen3-14b-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [14:17:44] RESOLVED: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=llm&var-backend=llm-qwen3-14b-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [14:58:00] (03CR) 10Sbisson: [C:03+2] Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) (owner: 10KartikMistry) [14:58:39] (03Merged) 10jenkins-bot: Update base image to Trixie [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1344129 (https://phabricator.wikimedia.org/T437213) (owner: 10KartikMistry) [14:59:25] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 9 - Sep 22]): Enable For You TTS experiment via remote config flag - https://phabricator.wikimedia.org/T438845#12406782 (10HNordeenWMF) 05Open→03Resolved [15:27:31] (03PS2) 10Hashar: decision-models: add clef model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1352454 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [15:36:06] 06Machine-Learning-Team, 10Ceph, 06Data-Platform-SRE, 06Infrastructure-Foundations, and 3 others: Design and build the next generation of container-registry service for the WMF production realm - https://phabricator.wikimedia.org/T413080#12407112 (10elukey) Updates from T437297 - I was able to drop ~400GB... [15:36:59] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12407124 (10Gehel) [15:37:37] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Decide on the future shape of ML/LW SLOs, and how they are monitored - https://phabricator.wikimedia.org/T434525#12407137 (10Gehel) [15:37:57] 06Machine-Learning-Team (Q1 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12407140 (10Gehel) [15:38:10] 06Machine-Learning-Team (Q1 FY2026-27), 10Prod-Kubernetes, 06ServiceOps, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), and 3 others: Update istio-cni to 1.29 - https://phabricator.wikimedia.org/T434198#12407138 (10Gehel) [15:38:24] 06Machine-Learning-Team, 10Cassandra, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 05Goal, 07OKR-Work: Provision Cassandra + Data Gateway resources for Tone Check - https://phabricator.wikimedia.org/T408129#12407143 (10Gehel) [15:39:03] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12407167 (10Gehel) p:05Triage→03Low [15:39:21] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Decide on the future shape of ML/LW SLOs, and how they are monitored - https://phabricator.wikimedia.org/T434525#12407176 (10Gehel) [15:40:01] 06Machine-Learning-Team (Q1 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE, and 2 others: eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12407213 (10Gehel) [15:40:09] 06Machine-Learning-Team (Q1 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE, and 2 others: eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12407217 (10Gehel) 05Resolved→03In progress [15:47:00] (03CR) 10Hashar: "recheck after having deployed the CI config change I24e485b35436727918d5ca0b4f3de3626c4b5570" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338218 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [15:53:11] 06Machine-Learning-Team, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): pytorch image builds are failing - https://phabricator.wikimedia.org/T437427#12407369 (10BTullis) [16:19:26] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work, 13Patch-For-Review: Add 'test/unit//' paths to Zuul trigger file filters - https://phabricator.wikimedia.org/T440288#12407727 (10hashar) Thanks for the task @gkyziridis ! @SBisson proposed a patch to add Zuul file filter to *som... [16:36:54] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Oct 07 - Oct 20]): TTS MVP within "For you" Data Validation - https://phabricator.wikimedia.org/T438506#12407916 (10Mazevedo) [17:06:52] 06Machine-Learning-Team, 10EditCheck, 06Editing-team, 06Growth-Team, and 4 others: Tone check: Improve handling of quoted content - https://phabricator.wikimedia.org/T426362#12408113 (10Sdkb) See T439643 for the newcomer tasks equivalent. [20:09:56] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Oct 07 - Oct 20]): TTS MVP within "For you" Data Validation - https://phabricator.wikimedia.org/T438506#12408950 (10SNowick_WMF) [20:10:26] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Oct 07 - Oct 20]): TTS MVP within "For you" Data Validation - https://phabricator.wikimedia.org/T438506#12408953 (10SNowick_WMF) Prod data passed for app version 50610-r-2026-10-06