[04:13:16] 06Machine-Learning-Team: Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11029883 (10ppelberg) [04:13:25] 06Machine-Learning-Team, 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11029885 (10ppelberg) [06:56:06] good morning. [07:22:12] good morning [07:40:59] morning folks! [07:57:14] folks, if you have pending reviews plz ping each other or drop a word in this channel to request someone to review [07:59:04] just mentioning it cause I noticed a couple of small patches pending [08:02:58] kevinbazira: bartosz can you coordinate the deployments for revertrisk so that we don't end up doing the same work twice? I'm referring to the kserve upgrade in revertrisk + the canonical wikis validation? I'm talking about these 2 patches https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1172011 , https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1171147 [08:04:34] isaranto: we are coordinating as shown in: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1172011/comment/78b4c9a9_36a09271/ [08:05:05] cool cool, you're one step ahead, thanks! [08:05:19] np! :) [08:06:01] just when I thought I was going to say something useful :P [08:06:29] it's very useful [08:08:15] Ah I should have set my patch to abandoned after we agreed to progress with Kevins patch, it makes sense that it got you confused Ilias, sorry for that! [08:08:58] np, nevermind me! [08:22:21] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install ml-serve101[2345] - https://phabricator.wikimedia.org/T393948#11030150 (10elukey) [08:23:02] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install ml-serve101[2345] - https://phabricator.wikimedia.org/T393948#11030153 (10elukey) [12:11:25] 06Machine-Learning-Team, 07Essential-Work: Upgrade langid model server from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400347 (10gkyziridis) 03NEW [12:12:11] 06Machine-Learning-Team, 07Essential-Work: Upgrade langid model server from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400347#11030776 (10gkyziridis) [12:13:14] 06Machine-Learning-Team, 07Essential-Work: Upgrade ores-legacy from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400348 (10gkyziridis) 03NEW [12:15:23] 06Machine-Learning-Team, 07Essential-Work: Upgrade articletopic-outlink model servers from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400349 (10gkyziridis) 03NEW [12:16:10] 06Machine-Learning-Team, 07Essential-Work: Upgrade revscoring model servers from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400350 (10gkyziridis) 03NEW [12:17:56] 06Machine-Learning-Team, 07Essential-Work: Upgrade article-descriptions model servers from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400351 (10gkyziridis) 03NEW [12:18:36] 06Machine-Learning-Team, 07Essential-Work: Upgrade reability model servers from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400352 (10gkyziridis) 03NEW [12:25:16] (03PS1) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [12:26:00] (03PS2) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [12:40:15] thanks for the reviews bartosz and aiko. [12:40:15] going to deploy the new RRLA and RRML images on staging ... [12:45:16] pods up and running [12:45:25] going to run load tests [12:55:45] Thank you Kevin for pushing this! <3 I'm very happy to help if something unexpected will come out [12:58:59] (03CR) 10Bartosz Wójtowicz: "Looks great, left just 1 small comment 😊" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:03:01] (03PS3) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [13:03:28] (03CR) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:04:30] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:04:40] (03CR) 10Gkyziridis: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:08:15] (03PS4) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [13:09:34] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:11:43] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install ml-serve101[2345] - https://phabricator.wikimedia.org/T393948#11031062 (10elukey) Reimaged ml-serve1013 with Trixie: ` [13:16:00] (03PS5) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [13:18:14] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:23:16] 10Lift-Wing, 06Machine-Learning-Team, 13Patch-For-Review: revertrisk model servers should return a 400 response for non canonical language names - https://phabricator.wikimedia.org/T399437#11031098 (10kevinbazira) Below are the load test results of the new RRLA and RRML images deployed on staging: ` Type... [13:27:02] (03PS6) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [13:29:09] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:36:29] (03PS7) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [13:38:28] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:41:29] (03CR) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [13:41:56] 10Lift-Wing, 06Machine-Learning-Team: revertrisk model servers should return a 400 response for non canonical language names - https://phabricator.wikimedia.org/T399437#11031178 (10kevinbazira) This is what I found in the RRML kserve logs on staging: ` 2025-07-24 13:37:06.071 1 kserve ERROR [errors.py:generic_... [13:45:42] (03PS8) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [13:45:45] 10Lift-Wing, 06Machine-Learning-Team: revertrisk model servers should return a 400 response for non canonical language names - https://phabricator.wikimedia.org/T399437#11031181 (10BWojtowicz-WMF) @kevinbazira As you said, it looks like we're failing due to missing NumPy dependency. It's indeed not defined in... [13:47:57] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [14:06:18] (03CR) 10Nik Gkountas: [C:04-1] "The implementation is promising and works well. Some suggestions for improvements." [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1171315 (https://phabricator.wikimedia.org/T399117) (owner: 10Sbisson) [14:20:06] (03PS9) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [14:22:13] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [14:35:13] (03PS10) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [14:37:38] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [14:42:56] (03PS11) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [14:46:20] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [15:11:27] (03PS12) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [15:13:00] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [15:18:03] (03PS13) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [15:18:53] 06Machine-Learning-Team, 06Research: Score probability evaluation for languages without enough data - https://phabricator.wikimedia.org/T398930#11031750 (10diego) He have develop a method to analyze languages without enough evaluation data. A detailed explanation can be found in [[ https://gitlab.wikimedia.org... [15:19:12] 06Machine-Learning-Team, 06Research: Score probability evaluation for languages without enough data - https://phabricator.wikimedia.org/T398930#11031751 (10diego) 05Open→03Resolved [15:19:29] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [15:21:04] (03PS14) 10Gkyziridis: revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) [15:23:27] (03CR) 10CI reject: [V:04-1] revertrisk-model: Update base image from bullseye to the latest bookworm image. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1172297 (https://phabricator.wikimedia.org/T400266) (owner: 10Gkyziridis) [15:29:34] georgekyz: can we work on resolving the autoscaling patch before mediawiki train is deployed in a couple of hours? [15:30:52] I'm available for the next 30' if you want to work on it together to figure it out [15:35:05] georgekyz: I went ahead and pushed something on top of your patch, hope you dont mind <3 [15:37:21] is anyone around for a review? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1171991 [15:48:45] +1 [15:48:48] I deployed the changes to staging & prod [15:49:02] and we're ready! [15:51:15] pods are initializing [15:53:24] new pods are up and running! [15:56:10] 06Machine-Learning-Team, 13Patch-For-Review: Configure autoscaling for tone-check model server - https://phabricator.wikimedia.org/T400162#11031970 (10isarantopoulos) 05Open→03Resolved We've deployed edit-check with the following autoscaling config: ` autoscaling.knative.dev/metric: "rps" auto... [16:43:29] * isaranto afk! [18:48:29] (03PS10) 10Sbisson: Recommendations based on difficulty level [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1171315 (https://phabricator.wikimedia.org/T399117) [18:51:19] (03PS11) 10Sbisson: Recommendations based on difficulty level [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1171315 (https://phabricator.wikimedia.org/T399117) [18:51:29] (03CR) 10Sbisson: Recommendations based on difficulty level (039 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1171315 (https://phabricator.wikimedia.org/T399117) (owner: 10Sbisson) [22:13:16] 06Machine-Learning-Team, 06Research, 05Goal: Goal: Apply the Tone Check model to published articles, to learn whether we can build a pool of high-quality structured tasks for new editors - https://phabricator.wikimedia.org/T392283#11033194 (10SSalgaonkar-WMF) [22:14:56] 06Machine-Learning-Team, 06Research, 05Goal: FY2025-26 Q1 Goal: Apply the Tone Check model to published articles, to learn whether we can build a pool of high-quality structured tasks for new editors - https://phabricator.wikimedia.org/T392283#11033198 (10SSalgaonkar-WMF) [22:15:33] 06Machine-Learning-Team, 06Research, 05Goal: Q1 FY2025-26 Goal: Apply the Tone Check model to published articles, to learn whether we can build a pool of high-quality structured tasks for new editors - https://phabricator.wikimedia.org/T392283#11033201 (10SSalgaonkar-WMF) [22:22:27] 06Machine-Learning-Team, 05Goal: Q1 FY2025-26 Goal: Enable volunteer evaluation of Tone Check model in additional languages - https://phabricator.wikimedia.org/T400423 (10SSalgaonkar-WMF) 03NEW