[07:29:36] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017), and 2 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#3595357 (10Alicia_Fagerving_WMSE) [13:48:31] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for Serbian - https://phabricator.wikimedia.org/T174687#3596724 (10Srdjan_m) [[ https://meta.wikimedia.org/w/index.php?diff=17213519 | Sorted ]]. [13:48:31] 10[4] 04https://meta.wikimedia.org/wiki/https://meta.wikimedia.org/w/index.php%3Fdiff%3D17213519 [15:00:24] o/ [15:42:54] halfak: when you have some time, could you ping me? I’d like to know how to get revscoring working on a stats machine so I can make some predictions for live & deleted revisons [16:37:03] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for Urdu - https://phabricator.wikimedia.org/T173190#3597552 (10awight) a:05awight>03None [16:47:24] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10Malayalam-Sites, and 2 others: Add language support for Malayalam - https://phabricator.wikimedia.org/T173193#3597640 (10awight) [16:51:18] o/ Nettrom [16:51:26] Hey dude. Just got done with morning meetings. [16:51:31] Let's get some revscoring going! [16:52:01] halfak: awesome! [16:53:15] First, virtualenv? [16:53:20] And which model do you want to run? [16:54:04] I’ll get a Python 3 virtualenv set up, I’ve already got your `venv` function and alias set up to active them [16:54:13] Nice! [16:54:18] I’m looking to run draftquality and wp10 [16:54:25] pip install wheel --upgrade [16:54:35] pip install pip --upgrade [16:55:16] pip install revscoring [16:55:16] aka “the secret sauce” [16:56:17] ooh, question [16:56:37] the stats machines don’t have much network access, do you build the virtualenv elsewhere and move it in? [16:57:01] https://wikitech.wikimedia.org/wiki/HTTP_proxy [16:57:03] :) [16:57:23] virtualenvs don’t like to be moved, unfortunately [16:57:23] aha! [16:57:28] I kinda figured that [16:57:32] but proxying is much better [17:00:16] I'm going to step away for lunch, but once you have that done, you should be able to just clone draftquality and wikiclass, install them and then use the models from the model dirs. [17:00:23] I'll be back in a bit to help if you run into trouble. [17:00:29] halfak: Picking up from our earlier chat, I haven’t decided whether the JADE scoring schemas should be discrete or not. [17:00:39] discrete? [17:00:52] halfak: no worries, I’ll ping you when I have the things installed [17:01:11] The strongest argument IMO for one “editquality” scheme which would integrate all the questions is that, for analysis, we want to see the form presented in exactly the same way, same order, with the same help text. [17:01:22] If any of those change, it should be reflected in the results data. [17:01:33] & we should be able to A/B test those changes. [17:01:45] can wait until after lunch though :D [17:02:33] The only thing I like about discrete schemas (goodfaith = 1 question = 1 schema) is that, if the value field is simple, then we can query it directly. [17:05:48] awight, +1. Maybe we can have a way to talk about a scope of a schema. E.g. damaging and goodfaith are "diff" scope. [17:06:44] * halfak --> lunch [17:07:12] That sounds right. & I suppose we should have something to allow only certain scoring schemas to be used, to reduce confusion. [17:08:07] e.g. if we’re trialing an editquality schema with modified help text, we don’t want people to arbitrarily use that... [17:58:06] halfak: I’m kinda nervous about running this, would like to check first: The new ORES cluster is deployed from tin, using scap? [17:59:09] awight: iirc i remember seeing it from tin yes [17:59:47] I can’t find the new boxes in any of the config files however—looks like deployments will still go out to the scb* boxes instead. [18:00:11] Amir1: would know awight [18:00:36] let me go check the sals [18:02:24] awight its done thru scb according to the last deployment [18:02:30] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:03:58] Zppix: ty, good thinking to check the SALs [18:04:04] np [18:04:17] So that makes sense, but I still don’t see how to deploy to the new cluster… continuing the hunt. [18:04:33] awight: the new cluster probably needs scap setup... [18:05:28] We have code deployed there already, with owner “deploy-service”, so someone did something. That’s all I know so far :D [18:06:49] akosiaris: question? do you have any idea if scap is setup for ores cluster yet and if not how could awight deploy to it? [18:48:21] wiki-ai/revscoring#1230 (master - 262b0ae : Adam Roses Wight): The build was broken. https://travis-ci.org/wiki-ai/revscoring/builds/274299630 [18:48:26] halfak: I see you’re about to do an ORES deployment? I have a deployment question for you above… [18:48:26] awight: I don't know what you meant by that [18:48:45] awight, we decided against it. :) [18:48:54] synja: o/. I was hoping you could mute that bot? [18:48:55] awight: Thanks! [18:49:26] * awight puts on best West Side Story face [18:49:59] :) [18:50:10] my client had all kinds of sFX for that [18:50:26] halfak: aha good. Can you show me how to deploy to the new cluster? [18:50:37] It looks like it gets deployed by scap from tin [18:50:46] but the config I found only deploys to scb*. [18:51:23] awight, when we deploy to the new cluster, we'll need to change that scap config. [18:51:38] okay, so it was deployed by ops livehack then? [18:51:42] If you are deploying what we have now, I think you can manually edit that config in order to get scap to do the right thing. [18:51:47] ooh [18:51:55] I think that the puppet did the initial install. [18:51:59] Maybe not [18:52:00] not sure. [18:52:15] well I wanted to stress test the one-liner that closes the model file. [18:52:25] * awight checks how merged that is [18:55:42] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017), and 2 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#3598253 (10Halfak) @Lokal_Profil, it looks like we have some delays. We'll need to finish T175053 fir... [18:56:02] wiki-ai/revscoring#1231 (filehandles - 5c54b00 : Adam Roses Wight): The build has errored. https://travis-ci.org/wiki-ai/revscoring/builds/274303104 [18:57:51] halfak: Argh. ^ That new CI failure is in the scaled threshold stats test. [18:57:59] https://travis-ci.org/wiki-ai/revscoring/builds/274299630 [18:59:57] WTF [19:00:03] I don't know why that failed. [19:00:11] 0.016 is < 0.05 [19:00:27] https://github.com/wiki-ai/revscoring/blob/master/revscoring/scoring/statistics/classification/tests/test_scaled_threshold_statistics.py#L75 [19:01:03] Oh! Something else. [19:01:07] The AUC is different [19:01:14] *PR AUC [19:01:31] That's a really big difference [19:01:35] 0.06 [19:10:28] aww awight. she was trying to learn and you silenced her [19:10:33] 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: UK dictionary broken in production - https://phabricator.wikimedia.org/T175627#3598302 (10Halfak) [19:13:18] 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: UK dictionary broken in production - https://phabricator.wikimedia.org/T175627#3598323 (10Halfak) Looks like 'aspell' is a better package. [19:16:02] 10Scoring-platform-team, 10ORES: Add LV dictionary to install. - https://phabricator.wikimedia.org/T175628#3598330 (10Halfak) [19:16:18] 10Scoring-platform-team, 10ORES: Add LV dictionary to install. - https://phabricator.wikimedia.org/T175628#3598343 (10Halfak) https://gerrit.wikimedia.org/r/377327 [19:18:12] wiki-ai/revscoring#1232 (ukwiki_dict - ed665ec : Aaron Halfaker): The build failed. https://travis-ci.org/wiki-ai/revscoring/builds/274310346 [19:22:12] 10Scoring-platform-team, 10revscoring, 10artificial-intelligence: UK dictionary broken in production - https://phabricator.wikimedia.org/T175627#3598354 (10Halfak) https://github.com/wiki-ai/revscoring/pull/359 [19:24:36] wiki-ai/revscoring#1234 (ukwiki_dict - d7072bd : Aaron Halfaker): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/274312318 [19:27:10] wiki-ai/revscoring#1236 (master - dc73edd : Adam Roses Wight): The build was fixed. https://travis-ci.org/wiki-ai/revscoring/builds/274313163 [19:30:20] halfak: Sorry about the delay, had a meeting too! But I’ve got revscoring installed on stats. Are there perhaps some additional things to install, e.g. nltk datasets? [19:30:47] Oh yeah. Check the readme.md for revscoring [19:30:53] There's a section for unix install. [19:30:57] https://github.com/wiki-ai/revscoring/blob/master/README.md#ubuntu--debian [19:31:02] will do, thanks! [19:31:04] Nice [19:32:18] halfak: I’m wondering about the best way to deploy the file handler patch on the new cluster, without leaving deployment out of sync. [19:32:44] awight, I think you'll want to edit the scap config on tin, do the deploy, then reset the tin repo. [19:32:49] I could cherry-pick that patch to the deployment branch, sync-file using a manually edited scap config file, then reverse the patch [19:32:59] Oh I like that better [19:33:02] halfak: ok sure. But what about the source. [19:33:08] "source"? [19:33:10] k, go with that plan? will do [19:33:14] :) [19:33:16] yeah there’s a revscoring change [19:33:42] Oh! Yeah, I'll get a patch together for you. Was just about to work on that. [19:33:58] hmm? [19:34:10] we'll have new deploy repo stuff soon. [19:34:43] I want to deploy just this change, https://github.com/wiki-ai/revscoring/commit/5c54b003222b2b48ff6b13a37e07864745bd2a38 [19:34:58] Do you want me to include other stuff for the purposes of stress testing? [19:35:20] awight, might as well merge it all. [19:35:25] did that change to revscoring get merged? [19:35:30] yes [19:35:45] cool. I'll have a new whl. [19:36:05] But we’re still deploying revscoring 1.x, so are there really backported fixes to deploy? [19:36:21] Ohhhhhhhh [19:36:25] halfak: ntlk stopwords downloaded, and it looks like stat1005 has all the necessary dictionaries installed [19:36:35] Nettrom, yeah. Should :) [19:37:02] awight, we could deploy revscoring 2.x [19:37:08] O_o [19:37:15] to the new cluster. [19:37:54] I could do that, but it’s probably best to make a new deployment directory if we’re playing that game. [19:38:14] Cos I wouldn’t want to be toggling back and forth between 1<->2 in the same directory [19:38:38] OK [19:38:42] Fair point. [19:39:00] So I propose to (1) create a backport of the fix as a branch of 1.3.x [19:39:23] (2) ssh to tin and build a bdist wheel of that [19:39:45] (3) remove whatever revscoring-1.3.*.whl is in the "wheels" submodule in the deploy and replace it with your wheel. [19:39:56] (4) profit [19:40:04] /deploy [19:40:16] cool. Happy to assume we aren’t going to roll that change back, it seems safe. [19:41:36] It’s mandatory to bump the version when releasing a new wheel? [19:41:55] nvm, I’ll do that for fun anyway. [19:42:06] only when uploading to pypi [19:46:06] awight, ^ aiming at master [19:46:12] oh I see you closed :) [19:46:15] ty, yep [19:46:35] * awight aims wand at own feet [19:47:46] nice. [19:48:00] wiki-ai/revscoring#1238 (filehandles_1.x - ac949cb : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/274321584 [19:49:16] awight, you can now pull the 1.3.19 wheel down from pypi :) [19:49:28] O_o [19:49:35] pip wheel revscoring==1.3.19 --no-deps [19:49:41] That’s weirdly easy [19:49:42] That will dump it into whatever your PWD is [19:49:44] :D [19:50:11] GH and pypi did that amongst themselves, or did you throw a lever? [19:51:02] I threw a lever [19:51:08] kk thanks [19:56:06] Having some submodule strangeness in https://gerrit.wikimedia.org/r/mediawiki/services/ores/deploy [19:56:25] It’s been a while, I must be forgetting an incantation [19:56:59] * halfak waits for specifics [19:57:21] I see git logs for lots of files in submodules/wheels, but my checkout only includes the revscoring .whl [19:57:35] ah nvm [19:57:50] git reset ; git checkout -f [19:57:50] FTW [19:58:46] halfak: `revscoring model_info` for the enwiki wp10 sure looks pretty! :) Say I want to score the quality of a deleted revision, can I chain some of the utilities to get that done? [19:59:56] Nettrom, do you know how to inject text into solve()? [20:00:20] https://github.com/wiki-ai/wikiclass/blob/master/wikiclass/utilities/score.py#L37 [20:00:27] This shows a good example ^ [20:00:35] halfak: Can you delete some of your branches there? https://github.com/wiki-ai/revscoring/branches/all [20:01:13] I updated my revscoring and running this thing for hrwiki again [20:01:30] now let's do other stuff 💪 [20:02:47] halfak: I’ll dig into that, thanks! [20:10:16] I’m creating a 1.3.x branch for our wheels deployment repo... [20:10:46] awight, Cool. When you submit a patchset, do "git review -R 1.3.x" [20:11:06] Nettrom, extra credit for updating the example usage of that function in the README for revscoring 2.0 :) [20:11:29] Nice, I didn’t know that shortcut. I’m usually added the branch in .gitreview#defaultbranch [20:11:29] basically, "from revscoring import ScorerModel" --> "from revscoring import Model" [20:14:57] awight, somewhat related https://gerrit.wikimedia.org/r/377333 [20:15:05] ^ includes your change in 2.0 [20:16:36] halfak: https://gerrit.wikimedia.org/r/377334 https://gerrit.wikimedia.org/r/377335 [20:17:20] halfak: In the past, we’ve self-merged library updates unless they’re tricky. What’s your preference? [20:18:01] I'm OK with it. [20:18:22] No not merge hhttps://gerrit.wikimedia.org/r/#/c/377335/ [20:18:25] Ity's aiming at master [20:18:36] Oh wait... Maybe that is OK [20:18:44] We haven't worked on 2.0 there yet [20:19:13] halfak: I should surely be able to do a bit of documentation updating. BTW, looks like the published version of the draftquality library doesn’t support the model in the GitHub repository. Is there a phab ticket for that? [20:19:40] Nettrom, that model should work fine. [20:19:43] halfak: That’s how it looked to me, yeah, the production deployment is all 1.3.x [20:19:47] Hmm... I have an updated one though. One minute. [20:19:55] +1 then awight. [20:19:59] gotta make some sammies… [20:19:59] kthx [20:20:08] halfak: the model is fine, but the library that gets installed through pip doesn’t have all the methods the mode expects [20:20:14] s/methods/functions/ [20:20:25] Nettrom, error message would help [20:20:42] halfak: of course, give me one second [20:21:15] Oh! I just remembered I'm blocked on model size. [20:21:20] It's over git's limit :/ [20:21:33] halfak: https://gist.github.com/nettrom/b8a4a3ceb2e9c35ca069b765b5357ea2 [20:22:03] Nettrom, are you running revscoring 2.0.6? [20:22:11] halfak: let me check [20:22:12] Oh! You need to install draftquality [20:22:45] from the base of draftquality "python setup.py install" [20:23:06] halfak: yes, I’m running 2.0.6 [20:23:08] also it looks like you have an old version of revscoring. [20:23:16] revscoring/scorer_models/scorer_model.py doesn't exist anymore [20:23:25] and yes, that’s what I expected as well, that I have to download the draftquality repo and install that, instead of using “pip install draftquality" [20:23:37] Nettrom, right [20:30:06] halfak: success! thanks for helping out with this, I’ll go check on some documentation and see where I can help update it. [20:30:23] Also fun to find out that apparently my first article creation would’ve been flagged as “spam” by the draft quality model, haha [21:38:13] halfak woant me to merge after i test https://github.com/wiki-ai/revscoring/pull/359 [21:41:35] Zppix, sure! [21:41:47] ok will run some tests halfak then merge [21:44:25] halfak: safe to del branch or no? [21:48:55] Zppix: yep, feel free to delete merged branches. Thanks! [21:50:56] Zppix: Would you mind checking to see if we’ve disabled the uk test in .travis.yml? [21:51:07] That package isn’t available in trusty [21:51:17] https://packages.debian.org/search?keywords=aspell-uk&searchon=names&suite=stable§ion=all [21:51:20] will look [21:51:27] if anything we can go to jessie cant we? [21:51:52] The restriction is, whatever releases Travis CI supports [21:52:21] awight: we can go up to jessie awight [21:52:26] just say the word and i can config it [21:52:46] What are we running in Travis now? [21:52:52] let me look [21:52:57] trusty [21:54:07] oh good point, I was looking at the wrong distro’s packages, entirely. Does Trusty have aspell-uk? Perhaps this was a good chase. [21:54:22] let me look [21:56:36] k I’ve deployed revscoring 1.3.19 with the RIIA filehandle fix, running a stress test now… [21:56:59] Using halfak’s bash history at https://phabricator.wikimedia.org/T169246#3481494 [21:57:02] Great! [21:57:03] Nice. [21:57:14] awight: it does infact [21:57:21] https://packages.ubuntu.com/trusty/allpackages [21:57:40] Zppix: Great! Nice due diligence [21:57:57] awight: travis wouldnt work all this time if it didnt :P [21:58:08] but double checking is always good [21:58:43] Zppix: good call—if there was already a uk test it would have fired just now before you merged [22:00:43] Here are the tests that we skip, https://github.com/wiki-ai/revscoring/commit/cc6e66ab20496a1b7082d51a076859473eb64393 [22:01:29] uk does use aspell. [22:01:30] hmm [22:01:48] Maybe aspell isn't so great on the travis image. [22:02:00] So, I’m surrounded by yellowjackets (North American for wasp), and a few flies. [22:02:12] halfak: what you thinking instead then? [22:02:21] Hard to keep track, hence I just swatted a wasp thinking it was a fly. [22:02:34] Scared of leaves rustling now [22:02:59] awight, eeek! [22:03:15] Zppix, seems we need docs on why we disabled certain tests on travis. [22:03:19] It's usually dict related. [22:03:23] It was a kind-hearted bump, probably unpleasant for the wasp but not vendetta-level it seems [22:04:03] halfak: In this case, I had started a goose chase to see if aspell-uk was included in the Travis-CI image. It was. [22:04:40] I think it's there but it doesn't work. [22:04:45] til, scap deploy -l "ores*.eqiad.wmnet" "Try ORES filehandle fix on new cluster" [22:04:50] Like myspell doesn't work on stretch or jessie [22:05:01] oooh [22:05:12] But uk is not in the skipped tests list, so Zppix reasons that its test is actually passing. [22:05:46] halfak: the tests are passing or travis would error out [22:05:51] https://gerrit.wikimedia.org/r/#/c/377345/ [22:06:51] Do you have permissions to run this on labs? T175627 [22:06:53] T175627: UK dictionary broken in production - https://phabricator.wikimedia.org/T175627 [22:07:03] Zppix: ^ if you’re interested [22:07:12] awight: i dont have access to the instances only to github [22:07:36] Do you happen to have ORES running locally? [22:08:11] It shouldn’t be too hard, there’s are relatively new mediawiki-vagrant roles. [22:08:52] yes i do [22:20:43] ORES response time, May 2017-today: https://grafana.wikimedia.org/dashboard/db/ores?panelId=15&fullscreen&orgId=1&from=1494116881137&to=1505168401137 [22:21:25] awight, interesting. Think we should investigate? [22:21:42] The more performant notch in mid-August is the one I’m most interested in learning more about. [22:21:50] Me too. Hmm. [22:23:17] 10Scoring-platform-team, 10Performance: Investigate ORES performant window in mid-August, 2017 - https://phabricator.wikimedia.org/T175640#3598766 (10awight) [22:24:23] Other than that, I’m not worried about the higher-latency windows. Just something to compare to the new cluster once we’ve switched over. [22:24:44] Agreed :) [22:25:21] This is a good example of metric we should keep long term :) [22:25:29] I'm out of here folks. have a good evening! [22:25:33] o/ [22:32:22] Score requests: [22:32:23] - errored: 23321 [22:32:23] - returned: 5049 [22:32:26] That doesn’t look right... [22:36:50] awight: if halfak is okay with it i have no problem being added to labs instance if you think its better [22:44:30] awight: do we want mediawiki/services/ores/deploy on our wikibugs config? [22:44:58] Zppix: yes, thanks I was thinking about that just today. [22:45:10] also the submodules. [22:45:45] awight: give me repo names and consider them added [22:47:10] Zppix: https://phabricator.wikimedia.org/source/ores-deploy/browse/master/.gitmodules [22:47:31] They’re funky URLs [22:47:36] ok [22:50:14] awight: https://gerrit.wikimedia.org/r/#/c/377364/ look good? [23:47:33] 10Scoring-platform-team, 10Analytics: Grafana has confusing or wrong scale for "scores errored" graph - https://phabricator.wikimedia.org/T175651#3599015 (10awight) [23:48:58] 10Scoring-platform-team, 10Analytics: Grafana has confusing or wrong scale for "scores errored" graph - https://phabricator.wikimedia.org/T175651#3599004 (10awight)