[04:55:39] (03Abandoned) 10Kosta Harlan: [WIP] Add AbuseFilter variable for revertrisk score [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1051837 (https://phabricator.wikimedia.org/T364705) (owner: 10Kosta Harlan) [06:03:37] 06Machine-Learning-Team: Build model training pipeline using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#10902721 (10kevinbazira) I have gathered answers to our initial planning questions, based on [Airflow Wikitech](https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow) docs.... [07:02:25] good morning! [07:07:20] Good morning [07:08:32] morning! [07:37:37] (03CR) 10Ilias Sarantopoulos: [C:03+2] improve logging logic for PopulateDatabase backfill script [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1154299 (https://phabricator.wikimedia.org/T395253) (owner: 10Gkyziridis) [07:39:30] good moening [07:45:24] don't know if you've seen this already https://github.com/apple/container [07:45:30] looks interesting [07:50:22] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 3 others: [batch #2] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395823#10902942 (10isarantopoulos) [07:53:13] Morning everyone [08:00:14] I'll be rebooting all of the staging machines today. There may be some disruption in serevices as we only have three workers (and only one with a GPU) [08:01:59] o/ Tobias! [08:02:03] ack, thank you! [08:05:24] isaranto: thnx for merging the backfill script [08:05:40] georgekyz: o a follow up from yesterday's discussion: this is the patch where the tone check model is now used https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1149663 [08:06:00] isaranto: niiice thnx for sharing! [08:17:31] OresUI enablement patch ready -> https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1155604 [08:44:37] 06Machine-Learning-Team, 13Patch-For-Review: Simplify pre-commit hooks within inference-services repository. - https://phabricator.wikimedia.org/T393865#10903158 (10BWojtowicz-WMF) [09:09:51] All hosts in staging have been rebooted and everything should be back to working order. Ping me if something's amiss. I will also start the same procedure in prod-codfw, but since we have more spare capacity there, there should be no disruptions. [09:21:11] klausman: after you wrote "prod-codfw" something in my brain triggered, a week ago when we completed the PSS migration for ml-serve-eqiad I didn't repool its discovery record [09:21:18] so we have been running only on codfw up to now [09:21:33] that is exactly what we do during the switchover week [09:21:36] ooh [09:21:37] but this time, not on purpose [09:21:44] really sorry, totally forgot about it [09:21:59] It happens. And you remembered, which is the important bit [09:22:07] I was convinced to have checked, but I didn't apparently [09:22:43] the long maintenance windows may end up in forgetting the last simple/meaningful test [09:22:45] On the positive side: apparently we can run single-site even when we don't want to. [09:23:05] that was already know/established during the switchover week, luckily [09:23:26] the thing that fooled me is having tested httpbb on both endpoints [09:23:33] not the discovery one [09:23:40] Well, during switchover week, everything else is in codfw as well, but now most prod stuff was in eqiad only, so at least there's that. [09:23:42] anyway, lemme know if anything weird pops up [09:23:55] ack, will do. [09:23:56] sure sure [09:28:24] 06Machine-Learning-Team: Build model training pipeline using WMF ML Airflow instance - https://phabricator.wikimedia.org/T396495#10903362 (10kevinbazira) I have worked on an initial stub for the ml training pipeline and tested in on the airflow dev instance. Below are the steps I followed: =====1.Create and pus... [09:29:19] I have created a stub for the ml training pipeline and tested it on the airflow dev instance. the steps I followed have been shared here: https://phabricator.wikimedia.org/T396495#10903362 [09:29:36] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 4 others: [batch #2] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395823#10903363 (10isarantopoulos) [10:22:57] georgekyz: we'll also need to schedule to deploy the patch for simplewiki and trwiki as well https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1151693 [10:23:11] we can deploy the 2 patches in the same window [10:29:03] isaranto: Alrighty, I will schedule both [10:29:58] can you review this: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1155604 in order to schedule it ? [10:30:23] yes [10:30:49] done! [10:32:52] scheduled both: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250612T1300 [10:35:01] thanks! [11:37:35] 06Machine-Learning-Team, 05Goal: Q4 24-25 Goal: Productionize tone check model - https://phabricator.wikimedia.org/T391940#10903947 (10achou) Update: * Collected recent newcomer's data, ran the model, and plotted probability distributions for 13 languages ([[ https://gitlab.wikimedia.org/repos/machine-learnin... [11:50:29] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, and 2 others: [batch #3] Enable revertrisk filters in recent changes in multiple wikis - https://phabricator.wikimedia.org/T395824#10904002 (10gkyziridis) a:03gkyziridis [12:26:53] I sent a patch to increase the number of workers for viwiki-reverted to see if it helps with the alerts we were getting https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1155655 [12:30:23] ack, +1 [12:31:02] thanks! [13:17:14] (03PS1) 10Sbisson: Add tests for page-collections and page-collection-groups [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1155666 [13:18:43] (03CR) 10CI reject: [V:04-1] Add tests for page-collections and page-collection-groups [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1155666 (owner: 10Sbisson) [13:31:12] I was reading https://docs.lmcache.ai/ from the kserve 0.15's press release, it looks interesting [13:32:18] I don't know why in the prerequisites cuda is listed [13:32:34] but they have a redis backend https://docs.lmcache.ai/kv_cache/redis.html [13:34:37] (03PS2) 10Sbisson: Add tests for page-collection-groups [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1155666 [13:36:02] TIL KV Cache, it is basically attention-layers-caching [13:36:05] very cool [13:36:37] (Ironically, Gemini explained it to me) [15:10:50] I merged the trwiki/simple wiki patch into this one so we will deploy a single patch tomorrow at 15:00 (whoever wants to feel the deployment adrenaline pings me :P ). [15:10:50] Final patch: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1155604 [15:11:29] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team, 13Patch-For-Review: [batch #1] Enable revertrisk filters in simplewiki & trwiki - https://phabricator.wikimedia.org/T395668#10905143 (10gkyziridis) I merged the patch for `trwiki` and `simplewiki` in... [15:27:56] georgekyz: I re-reviewed the previous patch and it seems that the threshold definitions from https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1151693 are missing [15:33:16] I can help with that since you're working with airflow etc [15:37:25] they are not highlighted but they are there. [15:40:19] I am not sure why is happened like this... probably because of the merging or something like that... but all the thresholds are set in this one: [15:40:20] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1155604?tab=comments [15:42:50] I figured it out. It is because this change consists of 2 commits [15:46:06] yeah I should probably needed to squash them [15:47:25] yep, we should do that. https://www.mediawiki.org/wiki/Gerrit/Tutorial#Squash_several_commits_into_one_single_commit_via_rebase [15:48:22] I pulled the one into the other and resolved the conflicts manualy... then I had the issue of gerrit that brings all the history of commits to the last one. [15:48:22] Bartosz and me we found two solutions: [15:48:22] ``` [15:48:22] git fetch gerrit [15:48:22] git rebase -i gerrit/main [15:48:22] ``` [15:48:23] And the other is just do: `git review -F` so just force it. [15:48:53] I pulled the one into the other and resolved the conflicts manualy... then I had the issue of gerrit that brings all the history of commits to the last one. Bartosz and me we found two solutions: [15:49:12] https://www.irccloud.com/pastebin/5G5dTv8j/ [15:50:11] or just `git review -F` so just force it. [15:50:17] isaranto: Should I squash it ? [15:52:22] georgekyz: yes you can follow the gerrit tutorial I shared [15:53:10] I'm available if you need help [16:00:47] Done: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1155604 [16:11:55] ty! [16:18:59] left one last nit, but we can deploy! [16:54:40] oh yeah great catch! [16:54:45] fixed already [18:53:13] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck, 10Editing-team (Tracking): Create SLO dashboard for tone (peacock) check model - https://phabricator.wikimedia.org/T390706#10906298 (10ppelberg)