[05:23:47] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), and 2 others: Manage ORES preferences on Watchlist (and Contributions) - https://phabricator.wikimedia.org/T160475#3142784 (10Etonkovidova) @SBisson Tested on cawiki-beta... [09:54:13] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ORES late march - https://phabricator.wikimedia.org/T160638#3143206 (10Trizek-WMF) >>! In T160638#3141813, @Halfak wrote: > https://ores.wmflabs.org/v2/scores/kowiki looks good IIUC, ko.wp has the predictions? If so, I may contact them. [09:56:34] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ORES late march - https://phabricator.wikimedia.org/T160638#3143208 (10Ladsgroup) kowiki is basic support (only reverted model) but hewiki has now damaging model (advanced support). [12:56:21] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), and 2 others: Manage ORES preferences on Watchlist (and Contributions) - https://phabricator.wikimedia.org/T160475#3143457 (10SBisson) >>! In T160475#3142784, @Etonkovidov... [13:10:51] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 06Operations, 06Services (designing), 15User-mobrovac: [spec] Active-active setup for ORES across datacenters (eqiad, codfw) - https://phabricator.wikimedia.org/T159615#3143490 (10akosiaris) Sorry for not answering sooner on this. @mobrovac That's an arch... [13:45:44] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ORES late march - https://phabricator.wikimedia.org/T160638#3143519 (10Trizek-WMF) >>! In T160638#3143208, @Ladsgroup wrote: > kowiki is basic support (only reverted model) but hewiki has now damaging model (advanced support). Ok, thanks. :) [14:56:25] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), and 2 others: Manage ORES preferences on Watchlist (and Contributions) - https://phabricator.wikimedia.org/T160475#3143696 (10Catrope) >>! In T160475#3143457, @SBisson wro... [16:57:19] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Conform ORES sensitivity levels to the new ERI standards - https://phabricator.wikimedia.org/T160575#3144192 (10SBisson) a:03SBisson [17:06:41] I'm trying to run revscoring locally, and it's not working very well for me [17:06:52] I ran pip install revscoring --user [17:06:53] Then: [17:07:03] https://www.irccloud.com/pastebin/y4HrQbWj/ [17:07:15] I'm not a Python dev so I'm way out of my depth here [17:07:16] Help? [17:07:26] ( Amir1 ? ) [17:07:49] RoanKattouw: Hey, let me take a look [17:08:21] RoanKattouw: You need to run it on python3 [17:08:28] At least the error indicates that [17:08:41] sudo pip3 install revscoring [17:10:05] Aha thanks [17:10:57] I'm around, just ping me when you encounter any issues [17:33:28] https://www.irccloud.com/pastebin/nk0MVvfx/ [17:33:31] Amir1: ---^^ [17:33:39] I tried pip3 install statistics --user but that didn't help [17:34:47] RoanKattouw: That's a lot of progress [17:35:01] You need to install statistics-related packages [17:35:10] let me see it's which library [17:37:20] RoanKattouw: Check if "pip3 install statistics" would fix it [17:39:00] Amir1: Requirement already satisfied (use --upgrade to upgrade): statistics in /home/catrope/.local/lib/python3.5/site-packages [17:40:24] Hmm for some reason it's using python2.7 in that backtrace [17:41:41] OK, installing statistics in python2 (using pip instead of pip3) worked [17:42:25] RoanKattouw: You are running it on python2 instead of 3, it will fail [17:42:35] It seems to be running fine now... [17:42:43] I thought the same thing [17:42:52] But surprisingly it's doing work and hasn't failed yet [17:42:57] the first part is fine (fetching labels) [17:43:12] but extracting features and building models/tuning report will fail [17:44:08] It hasn't failed yet so far [17:44:10] https://www.irccloud.com/pastebin/MVGFgJuF/ [17:45:27] That's... strange :D [17:45:33] Let's wait and see [17:46:34] How long does this normally take? [17:47:59] Depends on your network it might take up to one hour or so [17:48:22] (It does lots API requests) we usually this stuff inside stats machines [17:50:01] OK [17:50:11] Alternatively, is the output of that step hosted somewhere that I can download it from? [17:50:22] Because, let me explain what I'm actually trying to do [17:53:29] My understanding is that once I have plwiki.labeled_revisions.w_cache.resampled_15k_2016.json , I can run revscoring test_model plwiki.labeled_revisions.w_cache.resampled_15k_2016.json damaging -s 'recall_at_precision(min_precision=0.995)' for example [17:53:53] To get model stats that are not currently available from the ORES API, and explore them to decide which ones I would like to be available [17:54:27] You should be able to get that [17:54:33] but let me double check [17:59:28] RoanKattouw: It seems feasible. I can not say for sure until I test it myself but the data is inside of ores nodes (ores-compute-01) you can reextract it or tell me your commands and I run it for you [18:03:10] Oh it just finished building [18:03:27] But if I'm going to have to do all the others too, I'll see if I can get them from ores-compute01 instead [18:04:11] Awesome [18:04:45] Amir1: Could you add me to the ores project in labs? I can't access ores-compute01 otherwies [18:05:01] Yeah sure [18:06:13] RoanKattouw: Your LDAP user name? [18:06:19] Amir1: catrpoe [18:06:21] Sorry [18:06:23] catrope [18:06:24] thanks [18:07:09] RoanKattouw: Now you are admin and member, tell me if you need anything [18:07:13] Thanks [18:13:47] Amir1: OK, I'm in, and I've found a bunch of dataset/model files in /srv/halfak/projects/editquality/datasets [18:13:53] Is that where I should be looking? [18:14:05] Also, how do I run the revscoring command here? [18:14:19] probably you can find similar ones in ladsgroup path too [18:14:35] (mine is messy though\) [18:14:50] but you need to copy them inside your own home folder [18:14:51] Ah yes I see [18:14:55] OK [18:15:13] (also make a virtual env to have cleaner setup) [18:15:13] And then install revscoring using pip locally [18:15:13] ? [18:15:22] use virtualenv [18:16:32] OK, doing that now [18:18:16] Urgh, how do I make a virtualenv that uses python3? [18:18:34] --python=python3 it looks like [18:18:41] yeah [18:18:48] I do -p pytohn3 [18:18:50] *python [18:21:38] So now pip install revscoring builds scipy which fails with all sorts of C compilation erros [18:22:40] Cleaning my virtualenv and starting from scratch to see if that fixes it [18:22:56] Hah, it does [18:23:43] No wait, it worked in python2 [18:23:49] It fails spectaculraly in python3 [18:24:01] OK, plan B, I'll download the model file and run this locally [18:24:44] .... oh wait, I'm an idiot, I had the model file already :/ [18:24:47] It was in the git repo [18:40:28] Amir1: OK, so I got things working by downloading both the model file and the observations file from /srv/halfak, then running revscoring against them locally [18:40:39] But now I get results that are different from what https://ores.wikimedia.org/scores/plwiki/damaging/?model_info=test_stats says [18:41:06] Are the results non-deterministic? Or are the files in /srv/halfak not the files used in production? [18:42:12] RoanKattouw: It shouldn't be given that we do test on CV folding [18:42:32] For full context, this is what I ran: [18:42:33] but I think some randomness can happen [18:42:39] https://www.irccloud.com/pastebin/kvoGNTQf/ [18:42:49] Running the same thing again now to see if it outputs the same thing or something different [18:44:00] OK that did return the same [18:44:11] So I think I probably just don't have the same files that were used to generate the prod output [18:44:21] RoanKattouw: Check model version and compare it against prod [18:44:31] This one is 0.3.0 [18:44:49] https://ores.wikimedia.org/scores/plwiki/damaging/?model_info=test_stats is also 0.3.0 [19:01:45] I have no idea now, the data might differ or there is a huge bug somewhere [20:14:12] Amir1: OK now I've discovered that https://ores.wmflabs.org/scores/plwiki/damaging?model_info=test_stats and https://ores.wikimedia.org/scores/plwiki/damaging/?model_info=test_stats return different results and neither of them equal what I get locally :/ [20:15:24] I'm even more confused now [20:24:34] Looking through the infrastructure a bit more I'm starting to think that due to the place where things are randomized, it might not be possible to get stats after a model has been built, but only during the build process [20:27:31] Amir1: Random question: What does "filter rate" in "filter_rate_at_recall()" mean? Is it the same as precision, or does it mean something different? [20:28:10] And is fpr any different from 1-precision? [20:47:33] Hmm https://en.wikipedia.org/wiki/Precision_and_recall tells me that FPR is different and not immediately related to either precision or recall [21:01:24] 10Revision-Scoring-As-A-Service-Backlog, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Add more values to test_stats - https://phabricator.wikimedia.org/T161767#3145294 (10Catrope) It looks like filter_rate is not the same as precision. Is there a wa... [23:00:54] (03CR) 10Catrope: [C: 032] Don't make the "r" red except when highlighting is active [extensions/ORES] (wmf/1.29.0-wmf.18) - 10https://gerrit.wikimedia.org/r/345648 (https://phabricator.wikimedia.org/T161706) (owner: 10Catrope) [23:02:46] (03Merged) 10jenkins-bot: Don't make the "r" red except when highlighting is active [extensions/ORES] (wmf/1.29.0-wmf.18) - 10https://gerrit.wikimedia.org/r/345648 (https://phabricator.wikimedia.org/T161706) (owner: 10Catrope)