[08:04:50] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw: hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476 (10klausman) 03NEW [08:08:23] 06Machine-Learning-Team, 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw: hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478 (10klausman) 03NEW [08:08:49] 06Machine-Learning-Team (Q1 FY2026-27), 06SRE, 10SRE-swift-storage, 05Goal, 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12166214 (10MatthewVernon) [08:49:37] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166331 (10OKarakaya-WMF) prod locust latest test results to compare: https://phabricator.wikimedia.org/T418976#11703437 98.90 req/s 77 ms in 98% [09:21:00] (03PS1) 10Ozge: embeddings: Parameterize Locust test for multiple embedding models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319042 (https://phabricator.wikimedia.org/T432717) [09:33:04] (03CR) 10Kevin Bazira: [C:03+1] embeddings: Parameterize Locust test for multiple embedding models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319042 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [09:35:07] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166443 (10OKarakaya-WMF) Actually, the results are much better than I expected and slightly lower than current prod. Jina: - Jina: 99 percentile latency is... [09:52:43] (03CR) 10Ozge: [V:03+2 C:03+2] embeddings: Parameterize Locust test for multiple embedding models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319042 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [09:54:50] (03Merged) 10jenkins-bot: embeddings: Parameterize Locust test for multiple embedding models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319042 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:12:52] 06Machine-Learning-Team (Q1 FY2026-27), 06SRE, 10SRE-swift-storage, 05Goal, 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12166592 (10Ladsgroup) >>! In T432944#12163528, @MatthewVernon wrote: >... [10:22:00] (03PS1) 10Ozge: embeddings: Switch Jina embeddings to SentenceTransformers backend - Clean up. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) [10:22:54] (03CR) 10Ozge: "cleaning up the inference service as jina uses sentences transformers and not vllm." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:25:42] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166613 (10OKarakaya-WMF) [10:28:54] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166615 (10OKarakaya-WMF) [10:29:34] (03CR) 10Kevin Bazira: [C:03+1] embeddings: Switch Jina embeddings to SentenceTransformers backend - Clean up. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:30:16] (03CR) 10Ozge: [V:03+2 C:03+2] embeddings: Switch Jina embeddings to SentenceTransformers backend - Clean up. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:32:17] (03Merged) 10jenkins-bot: embeddings: Switch Jina embeddings to SentenceTransformers backend - Clean up. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1319056 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [11:18:14] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Serve jina-embeddings-v5-text-nano on Lift Wing - https://phabricator.wikimedia.org/T432717#12166767 (10OKarakaya-WMF) I've removed ml-serve1012 and interesting that the response time has increased to ~5 seconds. I'll set ml-serve1012 to see if this i... [12:57:49] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476#12167051 (10isarantopoulos) [12:57:58] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478#12167052 (10isarantopoulos) [12:58:51] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12167058 (10isarantopoulos)