[04:56:47] (03PS1) 10Kevin Bazira: decision-models: update clef MODEL_PATH default [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353727 (https://phabricator.wikimedia.org/T440223) [05:06:58] (03CR) 10CI reject: [V:04-1] decision-models: update clef MODEL_PATH default [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353727 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [05:26:16] (03PS2) 10Kevin Bazira: decision-models: update clef MODEL_PATH default [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353727 (https://phabricator.wikimedia.org/T440223) [06:55:45] (03CR) 10Ozge: [C:03+1] decision-models: update clef MODEL_PATH default [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353727 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [06:58:49] (03CR) 10Kevin Bazira: [C:03+2] decision-models: update clef MODEL_PATH default [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353727 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [07:00:46] (03Merged) 10jenkins-bot: decision-models: update clef MODEL_PATH default [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353727 (https://phabricator.wikimedia.org/T440223) (owner: 10Kevin Bazira) [07:16:24] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work, 13Patch-For-Review: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12414912 (10OKarakaya-WMF) let's test it on llm-qwen3-14b and we may move it to ml-serve1015 later. [07:59:44] 06Machine-Learning-Team (Q2 FY2026-27): Alert llm-qwen3-14b LiftWing service has a high rate of non 2/3/400 error code responses - https://phabricator.wikimedia.org/T440692 (10OKarakaya-WMF) 03NEW [08:05:30] 06Machine-Learning-Team (Q2 FY2026-27): Alert llm-qwen3-14b LiftWing service has a high rate of non 2/3/400 error code responses - https://phabricator.wikimedia.org/T440692#12415022 (10OKarakaya-WMF) We moved llm-qwen3-14b from ml-serve1015 to a larger partition (ml-serve1013/ml-serve1014) due to load. https://... [08:26:03] iridescence: o/ [08:35:43] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 05Goal: Setup MI300X nodes for LLM serving - https://phabricator.wikimedia.org/T424322#12415090 (10Gehel) [08:36:40] 06Machine-Learning-Team, 10Cassandra, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 05Goal, 07OKR-Work: Provision Cassandra + Data Gateway resources for Tone Check - https://phabricator.wikimedia.org/T408129#12415129 (10Gehel) [08:37:02] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Update kserve helm chart to include LLMInferenceService - https://phabricator.wikimedia.org/T433973#12415137 (10Gehel) [08:37:11] thanks for dropping by as well as using the models we are hosting! At this moment we are not offering models widely available to be used for agentic coding. Our initial goal is to enable tool builders to include LLM capabilities in their tools and we'll assess further adoption as we expand the platform and start onboarding more users [08:39:07] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Upgrade airflow-ml to 3.3.x - https://phabricator.wikimedia.org/T435481#12415183 (10Gehel) [08:39:55] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12415205 (10Gehel) [08:44:08] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Decide on the future shape of ML/LW SLOs, and how they are monitored - https://phabricator.wikimedia.org/T434525#12415293 (10Gehel) [08:44:18] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Remove article-descriptions model from LiftWing - https://phabricator.wikimedia.org/T438033#12415295 (10Gehel) [08:44:48] 06Machine-Learning-Team (Q1 FY2026-27), 10Prod-Kubernetes, 06ServiceOps, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), and 3 others: Update istio-cni to 1.29 - https://phabricator.wikimedia.org/T434198#12415305 (10Gehel) [08:45:12] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12415323 (10Gehel) [08:45:41] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12415331 (10Gehel) [08:46:17] 06Machine-Learning-Team, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Document procedure for updating the underlying distro for model servers (e.g. Bullseye -> Bookworm) - https://phabricator.wikimedia.org/T352189#12415347 (10Gehel) [08:46:23] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30): ml-lab: Update pytorch-rocm to a current ROCm-compatible release - https://phabricator.wikimedia.org/T428023#12415350 (10Gehel) [08:48:50] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12415410 (10Gehel) [08:49:05] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: monitoring: Grafana dashboard for LLM serving on MI300X - https://phabricator.wikimedia.org/T429237#12415412 (10Gehel) [08:49:35] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: Add the ML team to the POSIX group `docker` on `ml-build1001` - https://phabricator.wikimedia.org/T393566#12415421 (10Gehel) [08:50:54] (03CR) 10Gkyziridis: [C:03+2] revertrisk_wikidata: Develop KServe v2 and grpc inference protocol [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338218 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [08:51:56] (03Merged) 10jenkins-bot: revertrisk_wikidata: Develop KServe v2 and grpc inference protocol [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1338218 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [09:41:59] 06Machine-Learning-Team (Q2 FY2026-27): Deploy an open-weight decision model for classification use cases - https://phabricator.wikimedia.org/T440223#12415584 (10kevinbazira) We have deployed the [clef-flash](https://huggingface.co/Cloudflare/clef-flash) model in LiftWing prod under the llm namespace and it's ru... [09:57:30] (03CR) 10AikoChou: [C:03+1] benchmarks: add llm-qwen36-27b on a 96GB partition (ml-serve1013/1014) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353202 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [10:16:55] (03CR) 10Gkyziridis: [C:03+2] benchmarks: add llm-qwen36-27b on a 96GB partition (ml-serve1013/1014) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353202 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [10:18:11] (03Merged) 10jenkins-bot: benchmarks: add llm-qwen36-27b on a 96GB partition (ml-serve1013/1014) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353202 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [11:06:29] 06Machine-Learning-Team (Q2 FY2026-27), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: ml-serve: review documentation and apply AMD MI300X system-optimization settings to ml-serve1012-1015 - https://phabricator.wikimedia.org/T431553#12415821 (10achou) Baseline is done! I'm planning to run a... [11:10:25] 06Machine-Learning-Team (Q1 FY2026-27): Publish TTS v1 audio files with standard caching at the CDN - https://phabricator.wikimedia.org/T436758#12415831 (10kevinbazira) [11:14:31] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Audio quality evaluation for TTS v1: ASR round-trip benchmark and resulting fixes - https://phabricator.wikimedia.org/T433923#12415862 (10kevinbazira) 05Open→03Resolved Closing this task since we [published TTS v1 audio files](https://phabri... [11:15:31] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Pre-batch hardening for the TTS v1 experiment: generation-identity updates and the batch generation script - https://phabricator.wikimedia.org/T433594#12415873 (10kevinbazira) 05Open→03Resolved [11:22:51] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q1 FY2026-27 Text-to-Speech V1 Experiment - https://phabricator.wikimedia.org/T430676#12415902 (10kevinbazira) a:05isarantopoulos→03kevinbazira **Weekly Update:** The TTS audio is now in an A/B test on the Wikipedia app, with audio for 99.9... [11:24:23] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work, 13Patch-For-Review: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12415909 (10OKarakaya-WMF) ` MODEL=llm-qwen3-14b \ HOST_HEADER=llm-qwen3-14b.llm.wikimedia.org \ TOKENIZER=/home/ozge/benchmarks/qwen3-14b-fp8-token... [11:35:04] 06Machine-Learning-Team (Q1 FY2026-27): RevertRisk-Wikidata: move serving and stream to Linked Artifacts Cache - https://phabricator.wikimedia.org/T433699#12415968 (10gkyziridis) Tested revertrisk-wikidata on **ml-staging-codfw** from this [[ https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-se... [11:44:25] 06Machine-Learning-Team (Q2 FY2026-27): Define and implement SLOs for the public LLM inference services - https://phabricator.wikimedia.org/T440307#12415990 (10gkyziridis) ==== Qwen38-27b Benchmark on ml-serve1013/1014 96Gb partitions. ===== These are the results of the vllm-bench //(latencies in ms)//. More inf... [11:47:25] (03PS1) 10Gkyziridis: benchmarks: add llm-qwen38-27b on a 96GB partition (ml-serve1013/1014) experimental ns. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353920 (https://phabricator.wikimedia.org/T440307) [11:48:07] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work, 13Patch-For-Review: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12416003 (10OKarakaya-WMF) closing this ticket as we have confirmed that the configuration works and we have the following final values: - llm-qwen3... [11:48:54] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work: Update number of cpus in namespaces. - https://phabricator.wikimedia.org/T440575#12416004 (10OKarakaya-WMF) 05Open→03Resolved [11:50:14] 06Machine-Learning-Team (Q2 FY2026-27): Alert llm-qwen3-14b LiftWing service has a high rate of non 2/3/400 error code responses - https://phabricator.wikimedia.org/T440692#12416007 (10OKarakaya-WMF) 05Open→03Resolved [11:50:28] 06Machine-Learning-Team (Q2 FY2026-27), 07Essential-Work, 13Patch-For-Review: Horizontal scaling of LLM applications - https://phabricator.wikimedia.org/T440588#12416010 (10OKarakaya-WMF) 05Open→03Resolved [11:51:46] 06Machine-Learning-Team (Q1 FY2026-27): Hummingbird: Evaluate prompting and tooling quality, and identify future improvements - https://phabricator.wikimedia.org/T439206#12416012 (10OKarakaya-WMF) a:03OKarakaya-WMF [11:52:02] 06Machine-Learning-Team (Q1 FY2026-27): Hummingbird: Evaluate prompting and tooling quality, and identify future improvements - https://phabricator.wikimedia.org/T439206#12416014 (10OKarakaya-WMF) 05Open→03Resolved [11:52:51] (03CR) 10AikoChou: [C:03+1] benchmarks: add llm-qwen38-27b on a 96GB partition (ml-serve1013/1014) experimental ns. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353920 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:02:22] (03CR) 10Gkyziridis: [C:03+2] benchmarks: add llm-qwen38-27b on a 96GB partition (ml-serve1013/1014) experimental ns. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353920 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:02:50] (03Merged) 10jenkins-bot: benchmarks: add llm-qwen38-27b on a 96GB partition (ml-serve1013/1014) experimental ns. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353920 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:17:12] (03PS1) 10Gkyziridis: revertrisk_wikidata: Return INVALID_ARGUMENT for client errors on gRPC [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353935 (https://phabricator.wikimedia.org/T433699) [12:23:48] (03CR) 10CI reject: [V:04-1] revertrisk_wikidata: Return INVALID_ARGUMENT for client errors on gRPC [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353935 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [12:37:51] (03CR) 10Gkyziridis: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1353935 (https://phabricator.wikimedia.org/T433699) (owner: 10Gkyziridis) [13:03:44] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: LLM access request for OpenCode from the user (iridescence) - https://phabricator.wikimedia.org/T440265#12416177 (10OKarakaya-WMF) closing this for now as the request is replied. Looking forward to enabling it in the near future ` Ilias Sa... [13:04:13] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: LLM access request for OpenCode from the user (iridescence) - https://phabricator.wikimedia.org/T440265#12416181 (10OKarakaya-WMF) a:03OKarakaya-WMF [13:04:45] 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: LLM access request for OpenCode from the user (iridescence) - https://phabricator.wikimedia.org/T440265#12416184 (10OKarakaya-WMF) 05Open→03Resolved [13:35:53] 06Machine-Learning-Team (Q2 FY2026-27), 10Add-Link-Structured-Task, 06Growth-Team: Incorporate user decision signals into Add Link model training - https://phabricator.wikimedia.org/T435831#12416259 (10OKarakaya-WMF) Decisions on the Open Questions below: 1- We favor Row4 as it is easier to maintain and ther... [14:00:20] 06Machine-Learning-Team (Q2 FY2026-27), 06Language and Product Localization: Host sentence-transformers/LaBSE model in LiftWing - https://phabricator.wikimedia.org/T416659#12416345 (10isarantopoulos) [14:02:54] 06Machine-Learning-Team (Q2 FY2026-27), 06Language and Product Localization: Host sentence-transformers/LaBSE model in LiftWing - https://phabricator.wikimedia.org/T416659#12416357 (10isarantopoulos) Thanks for returning to this @santhosh. Is this model still the best candidate for low resource languages or ar... [15:56:19] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 23 - Oct 06]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12416803 (10HNordeenWMF) [15:57:44] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 23 - Oct 06]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12416811 (10HNordeenWMF) [17:50:06] 06Machine-Learning-Team (Q2 FY2026-27): Deploy an open-weight decision model for classification use cases - https://phabricator.wikimedia.org/T440223#12417351 (10isarantopoulos) Thanks for all the work! I added a row in the list of available models on Wikitech https://wikitech.wikimedia.org/wiki/Machine_Learning... [20:19:28] 06Machine-Learning-Team (Q1 FY2026-27), 06App Experience (AppEx Sprint [FY2627 Sep 23 - Oct 06]), 07Epic, 13Patch-For-Review: [Sub-epic] A/B test text-to-speech MVP within "For you" - https://phabricator.wikimedia.org/T434720#12418134 (10SNowick_WMF)