[14:30:05] o/ [14:59:44] I got stuck pushing LFS around yesterday. I'm picking up where I left off today with the prod config changes. [15:46:21] (03PS1) 10Halfak: General updates. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/489240 (https://phabricator.wikimedia.org/T214159) [15:46:44] Amir1, if you have some time now: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/ores/deploy/+/489240 [15:46:57] I think I might need to re-open our LFS mirroring task. [15:47:01] It seems like that is still broken. [15:50:28] what's wrong? let me check [15:51:19] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] General updates. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/489240 (https://phabricator.wikimedia.org/T214159) (owner: 10Halfak) [15:53:12] did you do "git submoduel sync" before? It fixed my local host [15:53:20] I'm still checking [15:57:37] halfak: I just checked in all of submodules and it worked fine [15:57:55] I just pushed all of the LFS to gerrit so that I could make my updates. [15:58:06] So I'm not surprised it worked for you [15:58:11] But I shouldn't have had to do that. [15:58:38] editquality and articlequality needed it. [15:58:57] Yeah [16:01:33] Alright! I think we're ready for a deployment. I'll do wmflabs first. [16:01:37] Then beta next [16:11:28] Amir1, see /home/hoo/draftquality/datasets/ on stat1007 [16:11:36] You can grab the "with_text" dataset [17:51:54] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Rebuild all models with revscoring 2.3.3 - https://phabricator.wikimedia.org/T215406 (10Ladsgroup) https://github.com/wikimedia/draftquality/pull/26 [18:13:29] Merged. [18:13:30] Amir1, it shouldn't make a difference for the models, but you might want to rebuild the next ones with revscoring 2.3.4 :) [18:13:40] Since that's what's ready for deployment right now. [18:13:57] I don't think rebuilding is worth the effort. [18:14:04] For models already built. [18:14:09] * halfak digs into editquality models. [18:15:08] I'm already at the middle of the last bit (drafttopic) [18:15:33] I couldn't do the articlequality, it's was erroring out on Russian [18:29:22] paste of the error? [18:31:59] https://www.irccloud.com/pastebin/SMflDyH0/ [18:33:57] Hmm. I've run into this before and I thought it was solved on stat1007. That's where you are training, right? [18:34:35] Mind if I experiment a bit with the ruwiki model? [18:35:02] yup [18:35:03] sure [18:38:59] Ok working on extracting now. [18:45:31] Amir1, O.O the whole process went to completion for me. [18:45:35] WTF [18:46:24] :/ [18:46:33] Should I just submit a PR in parallel for now? [18:46:48] yeah, I'm not doing anything on articlequality [18:48:17] Oh. haven't you rebuilt all of the other models? [18:50:00] yup, I did [18:50:02] https://github.com/wikimedia/articlequality/pull/76 [18:50:10] That's just the ruwiki model. [18:50:35] I'm going to go AFK for lunch soon but I'll be back and I hope to finish reviewing the editquality stuff (and anything left over) then. [18:52:00] wikimedia/articlequality#142 (revscoring-2.3.4_ruwiki - 747b7fd : Aaron Halfaker): The build passed. https://travis-ci.org/wikimedia/articlequality/builds/490684452 [19:00:10] Hmm. Looks like I'm seeing some consistent issues. In a few cases, our number of observations drops substantially. In other cases, revert detection is far more conservative than before. I think these are two independent weirdnesses that I've not seen before. [19:00:16] See my notes on the PR. [19:00:18] Amir1, ^ [19:00:23] OK off i GO! [19:00:55] okay [19:34:00] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Rebuild all models with revscoring 2.3.3 - https://phabricator.wikimedia.org/T215406 (10Ladsgroup) https://github.com/wikimedia/drafttopic/pull/30 [20:24:14] wikimedia/ores#1270 (docker_new_try - 06365cc : Amir Sarabadani): The build has errored. https://travis-ci.org/wikimedia/ores/builds/490720050 [20:42:30] Well that took way longer than expected. [20:42:32] Amir1, you still around? [20:42:53] yup, the articlequality is still in progress [20:43:14] Gotcha. Did you see my notes on editquality? [20:43:53] yeah and I have no explanation for that :/ [20:46:26] For the reverted edit proportion, I'd dig into the dataset itself. There's a few fields under "autolabel" that should be helpful. [20:47:00] For the datasets with fewer observations, I think looking for where the observations disappear would be good. [20:47:10] E.g. did they exist before extraction? [20:48:16] I think so [20:48:24] I'll check [20:51:37] Aha! it looks like we use 48 hour window and 3 edit revert radius. [20:51:47] I bet that is new and doesn't work as well for some of these wikis. [20:51:58] If they don't do quality control very actively than 48 hours may be too quick. [21:05:48] I go eat dinner, will be back soon [21:19:38] Looks like we're still dropping observations even when I remove those constraints. [21:23:05] Aha! Got it. Looks like the old model was trained with a dataset that contained duplicate observations. This is because of a sampling with replacement strategy we used to use. I think your models are good. I checked cswiki but not any of the others. Next I'll look into revert rates. [21:32:15] 10Scoring-platform-team, 10User-Ladsgroup, 10User-Zppix: Wiki-ai Travis-CI Image upgrade - https://phabricator.wikimedia.org/T183214 (10Ladsgroup) a:05Zppix→03Ladsgroup https://github.com/wikimedia/ores/pull/314 Going full docker [21:32:24] 10Scoring-platform-team (Current), 10User-Ladsgroup, 10User-Zppix: Wiki-ai Travis-CI Image upgrade - https://phabricator.wikimedia.org/T183214 (10Ladsgroup) [21:33:33] 10ORES, 10Scoring-platform-team (Current), 10Continuous-Integration-Config, 10User-Ladsgroup: Migrate ORES CI to Stretch - https://phabricator.wikimedia.org/T186239 (10Ladsgroup) a:03Ladsgroup https://github.com/wikimedia/ores/pull/314 Full docker [22:05:57] 10Scoring-platform-team (Current), 10User-Ladsgroup, 10User-Zppix: Wiki-ai Travis-CI Image upgrade - https://phabricator.wikimedia.org/T183214 (10Ladsgroup) 05Stalled→03Open [22:16:15] 10Jade, 10Scoring-platform-team (Current): Jade workshop position paper for HumBL - https://phabricator.wikimedia.org/T214960 (10awight) 05Open→03Declined Didn't have time to do this, regrettably. [22:37:13] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Implement better defaults for autolabel utlity - https://phabricator.wikimedia.org/T215671 (10Halfak) [22:37:52] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Implement better defaults for autolabel utlity - https://phabricator.wikimedia.org/T215671 (10Halfak) [22:37:55] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Rebuild all models with revscoring 2.3.3 - https://phabricator.wikimedia.org/T215406 (10Halfak) [22:38:47] wikimedia/articlequality#145 (json - b393e5a : Amir Sarabadani): The build failed. https://travis-ci.org/wikimedia/articlequality/builds/490774443 [22:41:12] <07EAAZEOI> wikimedia/articlequality#147 (json - cb65d4b : Amir Sarabadani): The build passed. https://travis-ci.org/wikimedia/articlequality/builds/490775473 [22:41:49] 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Rebuild all models with revscoring 2.3.3 - https://phabricator.wikimedia.org/T215406 (10Ladsgroup) https://codecov.io/gh/wikimedia/articlequality/pull/77 [23:05:27] I'm done for the day [23:05:30] o/