[00:11:11] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter): Add eswikibooks and svwiki to the beta cluster - https://phabricator.wikimedia.org/T188349#4004392 (10awight) [01:15:58] 10Scoring-platform-team, 10Global-Collaboration, 10MediaWiki-extensions-ORES: [Epic] Audit of pending ORES GUI deployments - https://phabricator.wikimedia.org/T187836#4004592 (10Catrope) [01:16:04] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10ORES, 10editquality-modeling, and 7 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#4004590 (10Catrope) 05Open>03Resolved a:03Catrope [01:16:19] 10Scoring-platform-team, 10Global-Collaboration, 10MediaWiki-extensions-ORES: [Epic] Audit of pending ORES GUI deployments - https://phabricator.wikimedia.org/T187836#3987540 (10Catrope) [01:16:23] 10Scoring-platform-team, 10Collaboration-Community-Engagement, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, and 3 others: Deploy ORES filters to Simple Wikipedia - https://phabricator.wikimedia.org/T182012#4004593 (10Catrope) 05Open>03Resolved a:03Catrope [01:17:00] *cough* [02:11:41] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 3 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#2131547 (10Catrope) I'm going to schedule this for deployment on Thursday March 1 a... [02:12:55] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 3 others: Deploy ORES filters in es.wikibooks - https://phabricator.wikimedia.org/T145394#2628616 (10Catrope) Scheduled for deployment on Thursday March 1 00:00-01:00 UTC, along... [07:45:10] 10Scoring-platform-team, 10Edit-Review-Improvements-Integrated-Filters, 10ORES, 10editquality-modeling, and 7 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#4005135 (10Lokal_Profil) [[https://sv.wikipedia.org/w/index.php?title=Wikipedia%3ABybrunnen&type=revision&diff=425... [08:41:40] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for nowiki - https://phabricator.wikimedia.org/T188235#4005209 (10jeblad) Still there is an editor that has to add those words to the article. Anyhow, if it is wrong write documentation! [10:15:43] (03CR) 10Alexandros Kosiaris: "I am thinking of merging this, any objections ?" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/409932 (https://phabricator.wikimedia.org/T171851) (owner: 10Alexandros Kosiaris) [10:21:18] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 3 others: Deploy ORES filters in es.wikibooks - https://phabricator.wikimedia.org/T145394#4005375 (10MarcoAurelio) @Catrope You can proceed. I'm not aware of any blockers here no... [10:23:20] 10Scoring-platform-team (Current), 10Packaging: Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#4005384 (10akosiaris) Why do we even need to create a debian package for shipping that single file ? I don't think it's worth the trouble. Can't we just ship it in the ORES re... [10:38:03] 10Scoring-platform-team, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 3 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#4005403 (10MarcoAurelio) @Catrope Unless @-jem- objects, I'm not aware of any block... [13:01:07] (03PS1) 10Zfilipin: Update README file for Selenium tests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/414997 (https://phabricator.wikimedia.org/T187862) [13:20:32] 10Scoring-platform-team (Current), 10Packaging: Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#4005825 (10Sumit) >>! In T187217#4001947, @awight wrote: > Working on the Debian packaging here: https://phabricator.wikimedia.org/source/word2vec/ > > @Sumit Is the gensim p... [13:25:51] (03CR) 10Ladsgroup: [C: 032] Update README file for Selenium tests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/414997 (https://phabricator.wikimedia.org/T187862) (owner: 10Zfilipin) [13:28:33] (03Merged) 10jenkins-bot: Update README file for Selenium tests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/414997 (https://phabricator.wikimedia.org/T187862) (owner: 10Zfilipin) [13:29:22] (03CR) 10jenkins-bot: Update README file for Selenium tests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/414997 (https://phabricator.wikimedia.org/T187862) (owner: 10Zfilipin) [13:41:18] wiki-ai/revscoring#1436 (w2v - 1b95e10 : Sumit Asthana): The build was broken. https://travis-ci.org/wiki-ai/revscoring/builds/346776644 [13:48:54] wiki-ai/revscoring#1438 (w2v - 64b4e17 : Sumit Asthana): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/346780638 [13:51:32] 10Scoring-platform-team (Current), 10Packaging: Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#4005891 (10akosiaris) >>! In T187217#4005825, @Sumit wrote: >>>! In T187217#4001947, @awight wrote: >> Working on the Debian packaging here: https://phabricator.wikimedia.org/... [14:45:08] o/ [14:47:48] 10Scoring-platform-team (Current), 10Edit-Review-Improvements-Integrated-Filters, 10ORES, 10editquality-modeling, and 7 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#4006144 (10Halfak) [14:48:19] Vermont, are the ORES filters working on the new recentchanges? [14:48:35] They’re in? [14:48:44] 10Scoring-platform-team (Current), 10Collaboration-Community-Engagement, 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, and 3 others: Deploy ORES filters to Simple Wikipedia - https://phabricator.wikimedia.org/T182012#4006146 (10Halfak) [14:48:51] Should be [14:49:01] Halfak: I still have that script for ORES enabled. Let me turn it off and see [14:49:06] :) [14:50:27] I seem to have misplaced the page where I had that script [14:51:51] Halfak: Found it. I’m on mobile and I’ll need to click the backspace button for every character. Could you kindly blank this for me? [14:51:53] https://simple.wikipedia.org/wiki/User:Vermont/sandbox1 [14:51:57] meta.wikimedia.org/User:/global.js [14:52:22] Can't Vermont [14:52:27] No rights for that. [14:52:31] Make me a simplewiki admin ;) [14:52:41] Just add "//" to the beginning of the line. [14:52:45] oh [14:54:44] Done. [14:54:48] It seems to be working. [14:54:56] \o/ [14:55:00] Sorry that took so long. [14:55:14] We recently scheduled a meeting with the Collab team (who work on the new edit filters UI) [14:55:23] And made sure they knew we were waiting on them. [14:55:29] And then they turned around really fast [14:56:22] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 3 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#4006203 (10Halfak) [14:56:41] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 2 others: Deploy ORES filters in es.wikibooks - https://phabricator.wikimedia.org/T145394#4006205 (10Halfak) [14:56:48] o/ awight [14:56:51] hey :) [14:57:03] You ever get that personal email from me? [14:57:43] The one about you being on a plane? [14:58:59] Oh the NAACL one? [15:00:05] yah that [15:00:48] I'm not sure how to respond. There's not much content in the paper. Are you planning to use revscoring/ORES for this project? [15:01:14] Yes that was the second question :) [15:01:19] We can chat later [15:01:28] I’ve just been having emails disappear, so wanted to check. [15:02:06] 10Scoring-platform-team (Current), 10Global-Collaboration, 10MediaWiki-extensions-ORES: [Epic] Audit of pending ORES GUI deployments - https://phabricator.wikimedia.org/T187836#4006237 (10Halfak) [15:02:17] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review: Add eswikibooks and svwiki to the beta cluster - https://phabricator.wikimedia.org/T188349#4006239 (10awight) The beta sites exist now, but I'm getting 404 when trying to save c... [15:02:19] :) OK [15:02:25] * halfak cleans up the boards a bit [15:02:32] I'll be looking at codezee's work next. [15:03:52] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review: Add eswikibooks and svwiki to the beta cluster - https://phabricator.wikimedia.org/T188349#4006254 (10awight) Beta svwiki has the damaging and goodfaith filters, eswikibooks cur... [15:03:58] 10Scoring-platform-team (Current), 10User-Ladsgroup: Edit quality campaign for Catalan Wikipedia - https://phabricator.wikimedia.org/T187771#4006255 (10Townie) We're done with this campaign as well. [15:04:04] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review: Add eswikibooks and svwiki to the beta cluster, enable ORES filters - https://phabricator.wikimedia.org/T188349#4006256 (10awight) [15:07:03] awight, should that be a sub-task of the audit? [15:07:10] Seems like maybe we need a beta audit [15:07:45] donno. I agreed to let it be decoupled from the deployments, so have no opinion on whether it belongs as a subtask. [15:07:51] Worse is: [15:08:26] grr can’t find a link. [15:08:41] 10Scoring-platform-team (Current), 10Edit-Review-Improvements-Integrated-Filters, 10ORES, 10editquality-modeling, and 7 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#4006291 (10awight) The filter labels have lots of untranslated strings, which is pretty bad. I would co... [15:08:43] ^ that, anyway. [15:10:53] Probably better to have the untranslated functionality that was requested than to take it away [15:11:00] 10Scoring-platform-team (Current), 10Edit-Review-Improvements-Integrated-Filters, 10ORES, 10editquality-modeling, and 7 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#4006298 (10awight) Here's a link to the untranslated strings: https://translatewiki.net/wiki/Special:Tra... [15:11:04] I'm sure many swedes have fallback languages that are apt. [15:11:12] English [15:11:54] Sure. [15:11:59] My understanding is that untranslated interfaces are one of the more offensive things we can be deploying. But yeah now that it’s deployed, we have a triage situation. [15:12:04] 10Scoring-platform-team (Current), 10Edit-Review-Improvements-Integrated-Filters, 10ORES, 10editquality-modeling, and 7 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#4006306 (10Halfak) Probably better to have the untranslated functionality that was requested than to tak... [15:12:09] tl;dr, it shouldn’t have been deployed w/o the translations. [15:12:35] I wasn't aware that this was so offensive. [15:14:06] K I could be wrong [15:14:09] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 3 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#2131547 (10awight) There are a few untranslated interface strings we need... [15:14:28] Ukranian falling back to Russian was a good show… [15:24:59] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 4 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#4006348 (10awight) [15:25:25] 10Scoring-platform-team (Current), 10Global-Collaboration, 10MediaWiki-extensions-ORES: [Epic] Audit of pending ORES GUI deployments - https://phabricator.wikimedia.org/T187836#4006351 (10awight) [15:25:36] 10Scoring-platform-team (Current), 10Edit-Review-Improvements-Integrated-Filters, 10ORES, 10editquality-modeling, and 8 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#4006349 (10awight) 05Resolved>03Open [15:30:49] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10ORES, 10Documentation: Elaborate documentation on how to deploy ORES to a new wiki - https://phabricator.wikimedia.org/T182054#4006364 (10awight) [15:32:15] akosiaris: Is this a good day to increase celery workers? I made my puppet patch a bit less aggressive. [15:32:53] akosiaris: lol you’re always a step ahead. I see that it was merged. [15:32:58] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 5 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#4006368 (10Trizek-WMF) [15:34:27] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 3 others: Deploy ORES filters in es.wikibooks - https://phabricator.wikimedia.org/T145394#4006372 (10Trizek-WMF) [15:36:03] akosiaris: Ah. The celery workers need to be restarted for those changes to take effect. I think you were saying it’s best to keep the restart dependency manual in this case? [15:36:14] I should have perms to do that, lemme see... [15:36:37] * awight secretly takes out `pssh` [15:44:27] akosiaris: Any idea? > ssh ores2001.eqiad.wmnet [15:44:32] channel 0: open failed: administratively prohibited: open failed [15:44:49] * awight facepalms. codfw. [15:48:31] done that a bunch of times [15:49:28] pssh is good stuff though. [15:49:33] I have a file with the hostsname, now [15:49:48] and can restart each server one at a time: pssh -p 1 -h ores-hosts sudo service celery-ores-worker restart [15:49:55] Impossible to misspell the servers :) [15:51:13] halfak: Take a peek at the recent available memory graph... [15:51:32] If I’m not reading it wrong, it says that celery workers are only taking up 10GB per node. [15:52:11] Cool. So we can up the count pretty safely. [15:52:13] * halfak digs a bit [15:52:13] and for some reason, 45 workers seems to use the same amount of memory as 35. We’ll know more after a few traffic surges. [15:52:19] Yeah, by a… lot [15:52:22] :-) [15:52:25] o/ [15:52:35] I can do that restart btw if you want [15:52:37] akosiaris: fyi, I’ve restarted celery on all ores* [15:52:38] it just slipped my mind [15:52:41] a ok [15:52:45] no worries! [15:52:55] I’m glad to get the commandline worked out. [15:52:55] awight, confirmed. Want to push the number of workers up a bit? [15:53:00] pssh ftw [15:53:06] Like, maybe double them and see what happens? [15:53:25] akosiaris: What do you think? You were sharing some philosophy about scaling services to need rather than to the machine’s max capacity. [15:55:06] yeah that's generally a more preferable approach. There's very rarely a need to push a box to its limits and it usually backfires, especially with modern equipment that features things like https://www.intel.com/content/www/us/en/architecture-and-technology/turbo-boost/turbo-boost-technology.html [15:56:06] we 've seen > 1 servers in that last 2 years suddenly clockstepping itself, due to some obscure in the BIOS code/setting about temperate vs cpu usage [15:57:02] Seems like a good argument for not going straight to max throttle, but we’re seeing that the ores* machines are downright *cold* [15:57:13] so the usual rule of thumb is "make sure you got enough resources to survive peak recorded traffic and add 20%" [15:57:35] to be able to survive spikes [15:57:52] akosiaris, that's great when choosing hardware. [15:58:04] so 45*9 = ~400 req/s given the avg req is ~1s [15:58:07] Right now, we're looking to allow ORES to optimally use the hardware we have. [15:58:23] my point is you are gonna be wasting your time [15:58:25] If we get a HUGE spike, it'd be nice to handle it. :) [15:58:29] feel free to do so [15:58:53] but it's almost definitely not worth your time, especially due to the law of https://en.wikipedia.org/wiki/Diminishing_returns [15:59:16] We do hit overload conditions at the current number of workers. [15:59:27] Uh. Seems like this isn't a total waste of time. [15:59:46] then this is something to be dealt with [15:59:50] Also, we tell our users to be a little slow on the requests because of our capacity. It would be cool to increase our recommendations. [16:00:14] E.g. right now, I say no more than 2 parallel requests for 50 scores at a time. [16:00:26] We could maybe bump that up to 4 parallel requests. [16:00:51] that's artificial (well not enforced really but anyway) limiting of natural traffic [16:01:00] https://grafana.wikimedia.org/dashboard/db/ores?refresh=1m&orgId=1&from=1519142441795&to=1519747181795&panelId=2&fullscreen [16:01:03] we obviously want as less of that as possible [16:01:13] but within reasonable limits ofc [16:01:24] akosiaris, we want researchers to use our service to score their stuff :) [16:01:24] we do have an API etiquette anyway [16:01:44] https://www.mediawiki.org/wiki/API:Etiquette [16:01:52] anyway just to be clear [16:02:01] I am not against raising the number of celery workers [16:02:04] * awight frowns at error graph which sometimes shows zero, sometimes shows the data [16:02:30] in fact we probably should raise them anyway [16:03:12] just don't go around spending too much time trying to figure the perfect number for that setting in order to say use 100% of the boxes capacity [16:03:20] +1 that sounds like good advice [16:03:21] How many workers per node were we running in stress testing? [16:03:31] akosiaris, yes. Of course. But close, would be nice :) [16:03:37] E.g. stop at 85% [16:03:47] even a bit less [16:03:53] https://phabricator.wikimedia.org/T169246 [16:03:56] Note, this is not load [16:04:00] it's capacity to handle load [16:04:14] ah that’s right, we were hitting a limit that seemed to be independent of hardware [16:05:57] We went up to 175 workers, hitting OOMs [16:07:57] awight, roger that. If we're at 45 workers now, maybe we could be at 100 and have plenty of head-room for adding more models and crap. [16:08:19] I’m combing through the mess in the task above, and in https://phabricator.wikimedia.org/T182249 [16:08:32] fwiw https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?from=now-3h&to=now&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=ores&var-instance=All says the boxes are not really doing much so feel free to increase the workers [16:09:04] even doubling them wouldn't probably hurt [16:09:34] halfak: These were the winning parameters during our stress test, https://gerrit.wikimedia.org/r/#/c/398316/2/hieradata/role/common/ores/stresstest.yaml [16:09:37] above that I would proceed a tad more cautiously. I do remember some open file limit within celery [16:10:07] 135 workers per node. +1 let’s just double for now, and let the dust clear. [16:10:16] 540 queue max size ? nice [16:10:42] Right, let's make sure to carry over the numbers on queue size and web workers too. [16:11:16] I had a formula for that somewhere. [16:12:21] btw, next quarter is coming. I was thinking of asking to be allowed to spend quite a lot of time working to migrate ORES to kubernetes [16:12:41] I 've hinted at that already [16:12:48] Do we have the business case for that written somewhere? Just cos I’m curious. [16:12:51] akosiaris, that sounds good to me. I'm happy to have ORES be an early convert. [16:12:56] awight, good Q. [16:13:02] anyway what this means is I might have time next quarter to help with a lot of things [16:13:06] :D [16:13:18] * akosiaris hopefully [16:13:44] halfak: looking for your formula, I see “bump the worker count to celery_workers + queue_size” [16:14:05] right. So I think we want to have the queue size be roughly # of workers / 2 [16:14:24] 9 * 90 (doubling celery workers) = 810 / 2 = 405 [16:14:51] 810 workers + 405 queue = 1215 web workers total / 9 = 135 [16:15:01] kk [16:15:34] For clarity: web workers (135), celery workers (90), queue_size (405) [16:17:59] https://gerrit.wikimedia.org/r/415032 [16:18:07] so in average 90 requests will have to use celery to get scored and 45 will be served from the cache ? [16:18:25] in average is wrong in this sentence I think [16:18:55] Essentially, half of our workers should be able to clear the queue. [16:19:30] fwiw, the proportion of requests served from cache dropped quite a bit recently [16:19:34] Essentially, we're willing to accept 1.5X our capacity before putting on the brakes. [16:19:51] awight, that's related to who is running an analysis. [16:19:54] We enabled more “precache” requests, scoring recent changes. [16:20:21] Looks like we're getting higher cache hit rates to me. [16:20:52] Looks like 85-95% except when someone is scoring a batch job. [16:21:47] cache hit rate excludes precaching. [16:21:55] We need to add a graph cos this is always confusing. [16:23:17] We’re currently at c. 1550/1700 = 90% *non*-cached scores [16:24:30] * awight adds a graph [16:26:31] awight, right. This is by design. [16:26:37] I'm confused what the problem is. [16:27:47] no problem. akosiaris was trying to draw a link between our new worker parameters and a real-world scenario, and I was just pointing out that we’re mostly serving uncached requests, 90% rather than the 1/3 he was hypothesizing. [16:28:11] But I’m making another graph because this has come up many times, so I want to just have a metric for % of scores coming from cache. [16:30:33] Oh I see! [16:36:46] ugh the difficulty of “1-x” in a fancy set-centric language [16:41:03] Well there it is, but it’s annoyingly the proportion of *uncached* responses which is confusing [16:49:03] done. [16:50:22] akosiaris: here’s a new graph that shows us the *absolute* cache hit rate, https://grafana-admin.wikimedia.org/dashboard/db/ores?refresh=1m&orgId=1&panelId=26&fullscreen [16:57:54] We should remove the Redis servers from the available memory graph, no? [16:58:01] They’re pinned at zero by design. [17:17:14] 10Scoring-platform-team (Current), 10artificial-intelligence: Build slide deck about AI at Wikimedia for Policy People - https://phabricator.wikimedia.org/T188124#4006744 (10Halfak) Slides were integrated. https://commons.wikimedia.org/wiki/File:Edit_review_AI_@_Wikimedia.pdf [17:22:51] gotta run for urgent thing [19:02:45] celery workers count bumped. no ill side effects yet [19:03:06] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [19:03:19] spoke too soon [19:08:55] seems like a spike [19:09:27] it has subsided already [19:12:07] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [19:42:41] The meetings stopped finally [19:42:42] lunch! [21:03:37] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 5 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#4007833 (10MarcoAurelio) @awight I'll take a look at those. [21:04:24] 10Scoring-platform-team (Current), 10Collaboration-Team-Triage, 10Edit-Review-Improvements-Integrated-Filters, 10editquality-modeling, and 5 others: Deploy ORES filters to Spanish Wikipedia - https://phabricator.wikimedia.org/T130279#4007838 (10MarcoAurelio) They've been all translated in the meanwhile. [21:22:17] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Implement word2vec featurevector in revscoring - https://phabricator.wikimedia.org/T188445#4007972 (10Halfak) [21:23:02] 10Scoring-platform-team (Current), 10Packaging: Package word2vec binaries - https://phabricator.wikimedia.org/T188446#4007994 (10Halfak) [21:23:15] 10Scoring-platform-team (Current), 10Packaging: Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#4008005 (10Halfak) [21:24:35] 10Scoring-platform-team (Current), 10ORES: Update ORES wheels for new revscoring requirements - https://phabricator.wikimedia.org/T188447#4008013 (10Halfak) [21:25:56] 10Scoring-platform-team (Current), 10revscoring, 10artificial-intelligence: Implement word2vec featurevector in revscoring - https://phabricator.wikimedia.org/T188445#4008026 (10Halfak) https://github.com/wiki-ai/revscoring/pull/391 [21:27:11] 10Scoring-platform-team (Current), 10Packaging: Support word2vec for production ORES models - https://phabricator.wikimedia.org/T187217#3968196 (10Halfak) Note that I've created {T188446} as a more specific sub-task regarding the packaging bits discussed above. [21:48:04] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/415133 (owner: 10L10n-bot) [21:49:54] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/415139 (owner: 10L10n-bot) [21:50:21] \o/ [21:53:09] (03PS1) 10Halfak: Bumps wheels for revscoring 2.2.0 [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/415165 [22:03:07] (03PS2) 10Halfak: Bumps wheels for revscoring 2.2.0 [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/415165 (https://phabricator.wikimedia.org/T188447) [22:07:18] halfak: I was travelling, I see you've merged word vectors patch, it looked good to me, I'll run a full CV with latest revscoring and word vectors to generate a report [22:07:26] *looks [22:07:57] Awesome! Yeah, I wanted to move forward and I figured there wasn't really anything controversial :) [22:09:04] halfak: if we have not yet broken anything, we might be very near to deploying drafttopic stuff ;) [22:09:19] \o/ [22:09:28] Next is going to be edit type. Because the model looks very similar [22:09:31] OneVsRest :) [22:09:48] I'm excited to try out the word2vec features in editquality too :) [22:10:25] yeah, sure, edit type is some new thing? after drafttopic? [22:10:34] i do recall reading the paper on it though [22:12:27] It did take me considerable time to come up with the decorators solution to handle aggregators without an API change, and now seeing it merged is :D [22:13:29] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Update ORES wheels for new revscoring requirements - https://phabricator.wikimedia.org/T188447#4008013 (10Halfak) a:03Halfak [22:13:48] edit type is actually old news. But yeah, we never went through the trouble of getting it working. [22:13:52] Now we can :) [22:13:59] Two birds, one PR [22:21:38] Good point on the load_kv() test [22:21:41] codezee, ^ [22:21:45] That's a complex one to test [22:24:24] no problem, i'll figure something out, lately i've played around with testing and mocking in python so i'm sure there'll be a way [22:43:35] hey halfak or Amir1: when you have a second, could you remind me: are there features do you deliberately decided not to include in the ORES edit quality models, even if they are strong predictors, because of concerns around inappropriate bias? [22:43:57] Yes. We don't have features for past warnings. [22:44:31] Adds a lot of signal, but also makes it so that once you receive one warning people the model won't leave you alone. [22:44:32] ah, yes, I remember that. [22:44:42] how about has_user_page? [22:44:57] I don't think we have that, but we could. [22:50:41] thanks, halfak [22:50:53] no problem :) [22:51:41] I'm working on a talk for UW on how to do AI stuff ethically and human-centered-ly, and ORES is featuring prominently. [22:51:50] \o/ [23:03:17] OK that's it for today. Have a good one folks.