[06:36:05] Good morning. [06:39:49] good morning! [06:50:22] good morning folks! [06:55:37] Hey, are the alerts above known issues or should we look into them? [06:58:57] the one related to reference-need is related to resources and scheduling as it can't find the available resources to schedule a pod. Not much of an issue. [07:01:11] the viwiki alert has been ongoing and I'll look into bumping the resources [07:01:33] I still need to plan our weekly ops rotation. we can discuss about this in our meeting and start it from next week [07:52:30] (03CR) 10Bartosz Wójtowicz: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1154242 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [08:01:46] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 3 others: [batch #2] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395823#10898297 (10isarantopoulos) [08:11:02] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 3 others: [batch #2] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395823#10898304 (10isarantopoulos) [08:11:35] ο/ georgekyz fyi I just ran the backfill script for bewiki (I updated the task as well) [08:30:31] good morning Folks [08:32:20] (03CR) 10Gkyziridis: "recheck" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1154299 (https://phabricator.wikimedia.org/T395253) (owner: 10Gkyziridis) [08:34:47] isaranto: Thnx for running the backfill for bewiki, I updated the script in for better logging. The number of 'successfully scored' is overwritten so what you saw in the logging is not the truth. [08:35:26] yes, I saw that thnx for fixing it [08:39:57] the tests are failling tho... but in files that I didn't touched... [08:44:19] good morning Aiko [08:47:29] (03CR) 10Gkyziridis: [C:03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1154242 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [08:55:33] morning! [09:05:05] georgekyz: you can open a bug report task on phabricator and paste the information from the failing tests. this would probably be caused by a recent merge in the extension repo (or a dependency). This should be fixed in a separate patch. If you can't fix that yourself you can look at git blame and ping whoever did that change and ask for help [09:07:42] you can find similar bug reports in phabricator (this is and old one I found related to the extension https://phabricator.wikimedia.org/T345922) [09:12:39] (03CR) 10Bartosz Wójtowicz: [C:03+2] ci: Add CI pipeline for pre-commit to be ran on entire repository. Add basic .dockerignore to the repo. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1154242 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [09:14:34] isaranto: thnx for sharing [09:17:29] I'll run the rest of the backfills in the tmux I have open. ok? [09:17:47] yeah sure [09:18:12] thnx a lot for running them, I am trying to understand what is failing over here [09:19:08] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 3 others: [batch #2] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395823#10898685 (10isarantopoulos) [09:25:33] (03Merged) 10jenkins-bot: ci: Add CI pipeline for pre-commit to be ran on entire repository. Add basic .dockerignore to the repo. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1154242 (https://phabricator.wikimedia.org/T393865) (owner: 10Bartosz Wójtowicz) [09:47:04] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 3 others: [batch #2] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395823#10898834 (10isarantopoulos) [09:49:05] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES: ORES Extension master branch is failing tests - https://phabricator.wikimedia.org/T396461#10898862 (10gkyziridis) [09:49:21] o/ TIL dags in this gitlab repo: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/tree/main/ml/dags [09:49:21] are what show up in our airflow ml instance: https://airflow-ml.wikimedia.org/home [09:49:54] Bug for ORES CI opened: https://phabricator.wikimedia.org/T396461 [10:02:36] kevinbazira: o/ yes! all the dags exist in that repo. we'll need to open MRs to add our dags there as well. For development, I forked the repo and added the dag to my fork, like https://gitlab.wikimedia.org/aikochou/airflow-dags/-/tree/main/ml/dags?ref_type=heads [10:03:05] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES: ORES Extension master branch is failing tests - https://phabricator.wikimedia.org/T396461#10898923 (10isarantopoulos) If I understand correctly it seems that these issues were introduced after this change https://gerrit.wikimedia.org/r/c/mediawiki... [10:04:00] kevinbazira: then I can test it in the airflow dev instance using https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/run_dev_instance.sh?ref_type=heads [10:14:55] 06Machine-Learning-Team, 06Security-Team, 07Security: Security Issue Access Request for Machine Learning team - https://phabricator.wikimedia.org/T396466#10898976 (10isarantopoulos) [10:59:58] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10ci-test-error (WMF-deployed Build Failure): ORES Extension master branch is failing tests - https://phabricator.wikimedia.org/T396461#10899112 (10A_smart_kitten) [11:05:16] aiko: o/ thanks for sharing [11:26:14] FYI, ml-etcd2003 is going down for a Ganeti reboot [11:31:05] aiko: did you use conda env in your local setup to install dependencies? Did you install airflow locally ? [11:43:09] oh sorry we running this in a stat machine [11:55:26] 06Machine-Learning-Team, 06Security-Team, 07Security: Security Issue Access Request for Machine Learning team - https://phabricator.wikimedia.org/T396466#10899251 (10Aklapper) Please make sure that everyone has set up 2FA by following https://www.mediawiki.org/wiki/Phabricator/Help#Multi-factor_authenticatio... [12:14:22] georgekyz: yes I used conda env in a stat machine. we don't need to install airflow in the conda env. The job logic is separated from the airflow scheduling logic [12:14:49] georgekyz: I set up the job repo https://gitlab.wikimedia.org/repos/machine-learning/edit-check following the doc here https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide/Python_Job_Repos [12:15:19] and published a conda env job artifact https://gitlab.wikimedia.org/repos/machine-learning/edit-check/-/packages [12:16:17] then the artifact is used in the airflow dag https://gitlab.wikimedia.org/aikochou/airflow-dags/-/blob/main/ml/dags/peacock_check_dag.py?ref_type=heads#L15 [12:32:12] aiko: thnx for the clarification. [12:57:22] np! the dev process can be tricky, so please don't hesitate to ask questions. I'll help with anything I know :) [12:57:54] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 3 others: [batch #2] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395823#10899523 (10isarantopoulos) [13:13:04] Folks, do we have any python lib for using s3cmd? For loading the model for instance from s3 similar to 'boto3' ? [13:15:55] o/ not that I know, we always used the s3cmd directly [13:16:17] what is the use case? [13:17:34] I am trying to build a retraining pipeline in airflow (for edit-check peacock) [13:18:49] So I was thinking either to use s3cmd via the `subprocess` python lib, or use `wget`to download it from https://analytics.wikimedia.org/ [13:26:52] I think you can probably use something like boto [13:27:18] we'll just need to instruct the Airflow Pods to run with the right AWS credentials as environment variables (me or Tobias can do it) [13:27:37] so you'll just use boto to pull from an s3 bucket etc.. [13:27:42] if should be possible IIRC [13:54:23] thank you [14:21:27] 06Machine-Learning-Team, 05Goal: Q4 24-25 Goal: Productionize tone check model - https://phabricator.wikimedia.org/T391940#10899990 (10isarantopoulos) [14:26:14] (03PS1) 10Máté Szabó: Set ORESDeveloperSetup to false by default [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1155247 (https://phabricator.wikimedia.org/T364705) [14:30:48] (03CR) 10Kosta Harlan: [C:03+1] Set ORESDeveloperSetup to false by default [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1155247 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [14:38:41] (03CR) 10CI reject: [V:04-1] Set ORESDeveloperSetup to false by default [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1155247 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [14:53:36] 06Machine-Learning-Team, 07sre-alert-triage: Alert in need of triage: DiskSpace (instance ml-lab1001:9100) - https://phabricator.wikimedia.org/T391465#10900103 (10klausman) 05Open→03Resolved a:03klausman SSDs have been enabled and 1002 is using Ceph homedirs. [14:53:46] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 3 others: [batch #2] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395823#10900106 (10isarantopoulos) a:03gkyziridis [14:54:07] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck: Create SLO dashboard for tone (peacock) check model - https://phabricator.wikimedia.org/T390706#10900107 (10isarantopoulos) [14:56:06] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Moderator-Tools-Team, 10Wikimedia-Extension-setup, 10Wikimedia-Site-requests: Install ORES extension on idwiki - https://phabricator.wikimedia.org/T382171#10900114 (10isarantopoulos) 05Open→03Resolved a:03isarantopoulos [14:57:48] 06Machine-Learning-Team, 06Language and Product Localization: Create a new S3 bucket for MinT - https://phabricator.wikimedia.org/T391958#10900138 (10isarantopoulos) 05Open→03Resolved [15:02:42] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck: Create SLO dashboard for tone (peacock) check model - https://phabricator.wikimedia.org/T390706#10900174 (10achou) [15:31:16] (03CR) 10Máté Szabó: "+2d I68e23931f64a9096b9299b891b9a62e22304cc94 which should fix CI." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1155247 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [15:48:34] 06Machine-Learning-Team: Build model training pipeline using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495 (10kevinbazira) 03NEW [15:49:13] As discussed in the meeting, I've created a phab task for building the model training pipeline using the WMF ML Airflow instance: https://phabricator.wikimedia.org/T396495 [15:49:13] Although they are not assigned to work on it, I've also added Özge and Bartosz as subscribers to this task since they shared valuable insights during the discussion on building this pipeline. [15:56:05] (03CR) 10Máté Szabó: "recheck" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1155247 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [16:05:07] (03PS6) 10Gkyziridis: improve logging logic for PopulateDatabase backfill script [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1154299 (https://phabricator.wikimedia.org/T395253) [16:05:34] (03CR) 10Ilias Sarantopoulos: "this should fix the CI issues https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1154761" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1154299 (https://phabricator.wikimedia.org/T395253) (owner: 10Gkyziridis) [16:05:37] 06Machine-Learning-Team, 10MediaWiki-Recent-changes, 10Moderator-Tools-Team (Kanban), 10MW-1.45-notes (1.45.0-wmf.4; 2025-06-03): [Spike] Investigate why filtering wasn't working on testwiki - https://phabricator.wikimedia.org/T395256#10900557 (10Kgraessle) 05Open→03Resolved [16:06:52] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10ci-test-error (WMF-deployed Build Failure): ORES Extension master branch is failing tests - https://phabricator.wikimedia.org/T396461#10900569 (10isarantopoulos) This should fix the issues https://gerrit.wikimedia.org/r/c/mediawiki/extensions/... [16:10:44] (03CR) 10Kosta Harlan: [C:03+2] Set ORESDeveloperSetup to false by default [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1155247 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [16:11:23] (03CR) 10Harroyo-wmf: [C:03+2] Set ORESDeveloperSetup to false by default [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1155247 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [16:25:20] (03Merged) 10jenkins-bot: Set ORESDeveloperSetup to false by default [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1155247 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [16:25:48] (03PS1) 10Máté Szabó: Set ORESDeveloperSetup to false by default [extensions/ORES] (wmf/1.45.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1155276 (https://phabricator.wikimedia.org/T364705) [16:26:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mszabo@deploy1003 using scap backport" [extensions/ORES] (wmf/1.45.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1155276 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [16:35:08] (03CR) 10CI reject: [V:04-1] Set ORESDeveloperSetup to false by default [extensions/ORES] (wmf/1.45.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1155276 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [16:37:28] (03PS2) 10Máté Szabó: Set ORESDeveloperSetup to false by default [extensions/ORES] (wmf/1.45.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1155276 (https://phabricator.wikimedia.org/T364705) [16:37:53] (03CR) 10TrainBranchBot: "Approved by mszabo@deploy1003 using scap backport" [extensions/ORES] (wmf/1.45.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1155276 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [16:50:27] (03Merged) 10jenkins-bot: Set ORESDeveloperSetup to false by default [extensions/ORES] (wmf/1.45.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1155276 (https://phabricator.wikimedia.org/T364705) (owner: 10Máté Szabó) [18:47:13] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10ci-test-error (WMF-deployed Build Failure): ORES Extension master branch is failing tests - https://phabricator.wikimedia.org/T396461#10901431 (10Umherirrender) 05Open→03Resolved a:03Umherirrender >>! In T396461#10900569, @isarantopo... [20:30:35] (03CR) 10Sbisson: [C:03+2] Add "page-collection-groups" endpoint [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1153311 (https://phabricator.wikimedia.org/T374695) (owner: 10Nik Gkountas) [20:30:43] (03CR) 10Sbisson: [C:03+2] Add support for fetching collection group recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1153312 (https://phabricator.wikimedia.org/T374695) (owner: 10Nik Gkountas) [20:32:04] (03Merged) 10jenkins-bot: Add "page-collection-groups" endpoint [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1153311 (https://phabricator.wikimedia.org/T374695) (owner: 10Nik Gkountas) [20:32:22] (03Merged) 10jenkins-bot: Add support for fetching collection group recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1153312 (https://phabricator.wikimedia.org/T374695) (owner: 10Nik Gkountas)