[06:39:35] 06Machine-Learning-Team, 07Essential-Work: Upgrade remaining model servers from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400144#11113705 (10kevinbazira) [06:44:08] 06Machine-Learning-Team, 07Essential-Work: Upgrade remaining model servers from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400144#11113712 (10kevinbazira) [06:58:05] good morning folks o/ [07:01:52] good morning! :) [07:20:40] good morning [07:21:16] 06Machine-Learning-Team, 07Essential-Work: Upgrade remaining model servers from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400144#11113758 (10kevinbazira) [07:31:34] morning morning o/ [07:31:35] going to deploy the new readability model-server in prod: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1180098 [07:43:52] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: Upgrade readability model server from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400352#11113823 (10kevinbazira) [07:44:31] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: Upgrade readability model server from debian bullseye to bookworm - https://phabricator.wikimedia.org/T400352#11113825 (10kevinbazira) The [new readability model-server](https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1180098) tha... [07:44:55] deployment completed --^ [07:55:20] georgekyz: I'm looking at defining the ml model training PVC. Just to make sure. Do you need concurrent access to the volume, or will it be mounted into a single pod at a time? [07:58:05] sorry, my bouncer went through a phase. That's brouberol [08:22:58] brouberol: I think it will be mounted on a single pod [08:23:17] ok, good to know. I'm submitting the patch right now then [08:24:20] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1181641 [08:28:15] Thank you very much [08:32:57] I +1 it [09:50:49] The PVC was created [09:50:49] airflow-ml-model-training-pvc Bound pvc-5456ec0d-ed4b-47be-a036-7ee46837cd49 20Gi RWO ceph-rbd-ssd 1s [09:56:17] brouberol: Thank you very much. I will follow your comments on the ticket. If there is anything missing, please update this ticket whenever you have time: https://phabricator.wikimedia.org/T396495#11101114 [09:57:21] Np! Actually, let me send you a patch real quick. I'm going to rename the PVC to remove the useless `-pvc` suffix [09:58:59] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1181658 [09:59:06] I've updated my comment to reflect the real PVC name [10:05:31] Thank you [10:07:23] all done [10:07:23] airflow-ml-model-training Bound pvc-d324c87c-e0ad-41de-8ed9-ad7095b4c4e7 20Gi RWO ceph-rbd-ssd 2s [10:07:25] np! [10:09:05] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking), 13Patch-For-Review: Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114241 (10brouberol) ` brouberol@deploy1003:~$ k get pvc airflow-ml... [10:58:03] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114371 (10gkyziridis) Thnx for deploying this @brouberol. Where this space actually exis... [10:59:28] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114385 (10brouberol) This exists in our Ceph cluster. If you want to read/write to it, yo... [11:39:13] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114481 (10gkyziridis) >>! In T396495#11114385, @brouberol wrote: > This exists in our Cep... [11:56:22] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114541 (10brouberol) I guess it depends on whether the csv is expected to change regularl... [12:40:51] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114697 (10gkyziridis) > If the CSV //is// expected to change regularly, I would probably... [12:51:13] 06Machine-Learning-Team, 06Growth-Team, 10Improve-Tone-Structured-Task, 05Goal, 07OKR-Work: Analyze samples of articles to see how many structured tasks we might be able to generate - https://phabricator.wikimedia.org/T401968#11114713 (10Michael) [13:06:23] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114776 (10brouberol) Tell me where I can find that CSV (on which host and which path) and... [13:27:13] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114843 (10gkyziridis) Model and Dataset: - [[ https://drive.google.com/drive/folders/1... [13:36:42] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114898 (10brouberol) Both drive links give me an access denied. Do you only need me to up... [13:50:50] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11114964 (10gkyziridis) [[ https://drive.google.com/drive/folders/1KUr-nuvE8p5kuYHFm_yFllI2... [15:15:07] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11115415 (10brouberol) I first copied the files onto the deployment server: ` $ scp Downloa... [15:17:23] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11115446 (10brouberol) Oh, should these have been located under `/mnt/model-training/tone-c... [15:20:52] georgekyz: the data has been transfered to the Ceph volume. I just have a final question in the ticket about the actual path you wanted, to make sure it's as you expected. [15:41:58] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11115541 (10gkyziridis) >>! In T396495#11115446, @brouberol wrote: > Oh, should these have... [15:47:37] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11115582 (10brouberol) Sure, I mounted the volume under `/mnt/model-training` and you seem... [16:20:53] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11115787 (10gkyziridis) Lets keep the second option: > ` > . # Ceph volume mountpoint > └─... [17:00:14] 06Machine-Learning-Team, 10Data-Platform-SRE (2025.08.16 - 2025.09.05), 10Editing-team (Tracking): Build model training pipeline for tone check using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#11116023 (10brouberol) ` airflow@ml-copy-data:/mnt/model-training$ find . . ./training ./tr... [21:19:33] 06Machine-Learning-Team, 10EditCheck, 10VisualEditor, 10Editing-team (Planning), 07Epic: Expand language coverage for Tone Check - https://phabricator.wikimedia.org/T394448#11117033 (10ppelberg)