[07:24:24] 06Machine-Learning-Team, 06Collaboration-Services, 10GitLab, 06Product Safety and Integrity, and 4 others: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org - https://phabricator.wikimedia.org/T437216#12383683 (10ABran-WMF) [08:36:06] 06Machine-Learning-Team (Q1 FY2026-27), 06Collaboration-Services, 10GitLab, 06Product Safety and Integrity, and 4 others: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org - https://phabricator.wikimedia.org/T437216#12383821 (10isarantopoulos) [08:42:54] 06Machine-Learning-Team (Q1 FY2026-27), 10Article-Guidance, 10LPL Projects (Guidance while editing - FY27 DE 1.4.4), 10LPL sprints (S3 2026 September), and 2 others: Spike: scope work needed to enable single domain lookup in the existing Reference Risk mod... - https://phabricator.wikimedia.org/T435491#12383828 [09:53:10] 06Machine-Learning-Team (Q1 FY2026-27): Evaluate LibreChat and Open WebUI for LiftWing Studio Productionization - https://phabricator.wikimedia.org/T438987#12384183 (10achou) We've also settled on the name for the user-facing UI: **Hummingbird Chat** Hummingbird is the name of the LLM project, so we'll use this... [10:07:54] (03CR) 10Gkyziridis: "Thnx for working on this Stephane." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) (owner: 10Sbisson) [10:22:24] (03CR) 10Gkyziridis: reference-risk: accept a single domain as input (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) (owner: 10Sbisson) [10:52:58] 06Machine-Learning-Team (Q1 FY2026-27), 10Add-Link-Structured-Task, 06Growth-Team: Incorporate user decision signals into Add Link model training - https://phabricator.wikimedia.org/T435831#12384391 (10OKarakaya-WMF) **I'll add my notes soon!** enwiki: {F105699195, width=500} [11:04:39] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Move llm isvcs to new partitions - https://phabricator.wikimedia.org/T439555#12384403 (10gkyziridis) ==== qwen36-27b Deployed on ml-serve1013/1014 ==== Sharing the 96GB partition size. `lang=Shell $ kube_env llm ml-serve-eqiad $ kubectl get pods NAME... [11:04:43] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Move llm isvcs to new partitions - https://phabricator.wikimedia.org/T439555#12384405 (10gkyziridis) [11:39:48] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Move llm isvcs to new partitions - https://phabricator.wikimedia.org/T439555#12384514 (10gkyziridis) ==== qwen38-27b Deployed on ml-serve1013/1014 ==== Sharing the 96GB partition size of the `ml-serve1013/1014` machines. `lang=Shell $ kubectl get pods NAME... [11:39:51] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Move llm isvcs to new partitions - https://phabricator.wikimedia.org/T439555#12384515 (10gkyziridis) [12:43:51] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Move llm isvcs to new partitions - https://phabricator.wikimedia.org/T439555#12384826 (10gkyziridis) `lang=Shell ` [12:47:19] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Move llm isvcs to new partitions - https://phabricator.wikimedia.org/T439555#12384842 (10gkyziridis) [14:08:55] (03PS2) 10Sbisson: reference-risk: accept a single domain as input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) [14:15:53] (03CR) 10Sbisson: "Thanks! I've addressed both points and made a config patch to run the tests in CI (I2585afcf574d792723f10cc87364d182ad39dceb)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) (owner: 10Sbisson) [14:49:27] (03CR) 10Hashar: reference-risk: accept a single domain as input (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) (owner: 10Sbisson) [14:51:41] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 23 - Oct 06]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12385694 (10HNordeenWMF) [14:51:43] (03CR) 10Gkyziridis: [C:03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) (owner: 10Sbisson) [14:51:54] (03CR) 10Gkyziridis: [C:03+1] "Done" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) (owner: 10Sbisson) [15:05:30] (03PS3) 10Sbisson: reference-risk: accept a single domain as input [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) [15:42:47] ml-team [15:42:55] who is there [16:39:31] (03CR) 10Sbisson: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) (owner: 10Sbisson) [16:39:59] (03CR) 10Hashar: "recheck after having deployed I2585afcf574d792723f10cc87364d182ad39dceb *Add the reference-quality test pipeline*" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1344692 (https://phabricator.wikimedia.org/T435491) (owner: 10Sbisson) [17:22:44] FIRING: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=llm&var-backend=llm-qwen3-14b-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [18:23:56] Hey PCendr! [18:52:44] RESOLVED: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=llm&var-backend=llm-qwen3-14b-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate