[01:49:14] (03CR) 10Eamedina: [C:03+2] Update user-agent contact information [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1287015 (owner: 10Sbisson) [01:49:50] (03CR) 10CI reject: [V:04-1] Update user-agent contact information [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1287015 (owner: 10Sbisson) [06:14:47] (03CR) 10Ilias Sarantopoulos: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) (owner: 10Ilias Sarantopoulos) [07:04:57] (03CR) 10KartikMistry: [C:03+2] "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1287015 (owner: 10Sbisson) [07:06:27] (03Merged) 10jenkins-bot: Update user-agent contact information [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1287015 (owner: 10Sbisson) [07:37:41] (03CR) 10Kevin Bazira: "I see CI added -1" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) (owner: 10Ilias Sarantopoulos) [08:14:36] (03PS10) 10Ilias Sarantopoulos: qwen36-27b: add model server for Qwen 3.6 FP8 inference [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) [08:16:55] (03CR) 10Ilias Sarantopoulos: "It was failing because it was also trying to build the production image which will likely run out of space." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) (owner: 10Ilias Sarantopoulos) [08:32:41] (03CR) 10Kevin Bazira: [C:03+1] "great. LGTM. test it on the experimental ns." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) (owner: 10Ilias Sarantopoulos) [08:36:25] (03PS11) 10Ilias Sarantopoulos: qwen36-27b: add model server for Qwen 3.6 FP8 inference [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) [08:51:45] (03PS12) 10Ilias Sarantopoulos: qwen36-27b: add model server for Qwen 3.6 FP8 inference [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) [09:09:28] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#11920747 (10isarantopoulos) I've implemented the streaming responses by using the OpenAIChatAdapter. Although this works locally I'm pretty sure that... [11:40:28] (03CR) 10Ilias Sarantopoulos: [C:03+2] qwen36-27b: add model server for Qwen 3.6 FP8 inference [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) (owner: 10Ilias Sarantopoulos) [11:48:43] (03Merged) 10jenkins-bot: qwen36-27b: add model server for Qwen 3.6 FP8 inference [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1285375 (https://phabricator.wikimedia.org/T425680) (owner: 10Ilias Sarantopoulos) [11:52:57] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11921322 (10isarantopoulos) I've reached out to #wikimedia-serviceops to get reviews for... [12:30:35] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11921409 (10isarantopoulos) @DPogorzelski-WMF & @klausman please also coordinate with #wi... [13:04:03] 06Machine-Learning-Team (Q4 FY2025-26), 06Traffic, 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11921513 (10isarantopoulos) [13:08:12] 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 06Traffic, 13Patch-For-Review: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#11921528 (10isarantopoulos) [16:20:38] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review, 10Toolforge (Quota-requests): Request increased quota for wiki-tts Toolforge tool - https://phabricator.wikimedia.org/T425804#11922494 (10taavi) 05Open→03Resolved a:03taavi [16:20:53] 06Machine-Learning-Team (Q4 FY2025-26), 10Cloud-VPS (Project-requests): Request creation of wikitts VPS project - https://phabricator.wikimedia.org/T425909#11922514 (10taavi) 05Open→03Declined Per {T425804} happening instead. [16:43:18] (03PS1) 10Alex Paskulin: openapi: Copyedits and style fixes [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1287445 (https://phabricator.wikimedia.org/T419455) [16:52:56] (03CR) 10Alex Paskulin: openapi: Copyedits and style fixes (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1287445 (https://phabricator.wikimedia.org/T419455) (owner: 10Alex Paskulin) [17:01:39] 06Machine-Learning-Team, 10ORES, 10AntiSpoof, 10BetaFeatures, and 2 others: Drop extensions from closed wikis where the database tables are unused - https://phabricator.wikimedia.org/T420052#11922800 (10Dreamy_Jazz) [20:34:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [20:34:55] Deployment qwen36-27b-predictor-00001-deployment in experimental at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=experimental&var-deployment=qwen36-27b-predictor-00001-deployment - ... [20:34:55] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [20:44:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [20:44:49] Deployment qwen36-27b-predictor-00001-deployment in experimental at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=experimental&var-deployment=qwen36-27b-predictor-00001-deployment - ... [20:44:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [21:09:16] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#11923434 (10isarantopoulos) The initial attempt for this deployment unfortunately fails horribly due to the missing support for this architecture from... [23:50:12] 06Machine-Learning-Team, 10EditCheck, 06Growth-Team, 10Revise-Tone-Structured-Task: Tone check: Improve handling of quoted content - https://phabricator.wikimedia.org/T426362#11923993 (10KStoller-WMF)