[15:00:24] o/ [16:04:33] o/ Amir1 [16:04:38] Back to the normal work pattern today [16:05:05] I'm hoping to do some mopping around the whole ecosystem over the next couple of days. [16:05:45] I'd like to deploy the new models soon, but we need to coordinate that deployment with patches to Huggle/ScoredRevisions/etc. [16:05:58] ^ That's my goal for the week. [16:06:12] Well... that can kick off the second pilot for Edit types. [16:06:18] I'm almost done with that though. [16:20:31] Welp, that took out the labels server [16:20:33] Darn it. [16:20:38] Why didn't this happen on staging [18:05:57] halfak: I just woke up [18:05:58] sorry [18:06:45] Hey! No worries. [18:06:53] Just ping'd you because you were logged in :) [18:07:32] I forgot to suspend the lap top [18:08:47] anyway [18:08:47] what can I do? [18:10:58] halfak: I also updated the introduction in meta wiki but it's really tedious [18:11:28] I was thinking if we can get a person from operations on board [18:11:30] Amir1, help me make a list for spring cleaning. [18:11:40] What *should* we have? What needs attention. [18:11:56] Lets gather those things and prioritize. [18:12:03] I'll be digging into that in a couple of hours. [18:12:11] Just about to head out to lunch and then I have some meetings. [18:12:24] what do you mean by spring cleaning? I'm still loading [18:14:10] E.g. we have a large set of models and revscoring 1.0 that isn't yet deployed. [18:14:50] We have a lot of proposals for utilities we want (e.g. a profiling feature extractor) that haven't been picked up yet. [18:14:57] These aren't collected in a list in a nice way yet. [18:15:13] I get it [18:16:23] so basically you want a to-do list? [18:17:31] Yeah. Specifically go looking for stuff to do. [18:17:33] :) [18:18:30] sure, I just got an interesting email about the deployment [18:19:20] Amir1, will be back in a bit. [18:19:27] ok :) [18:19:30] Please feel free to dump thoughts in scrollback and I'll read when I get back [18:19:47] sure [18:29:04] OK. I think we should classify our work into several categories: 1- revscoring 2- ores 3- the extension [18:29:51] in revscoring I think we need 1- semi-supervised support 2- support more languages 3- profiling 4- the json support [18:30:05] in ores we need 1- deploy ores into prod [18:31:55] in the extension we need to work based on community feedback after deployment (specially I'm looking forward to seeing this extension in wikidata) [18:32:38] I think we also need to build an article quality model and then use it somewhere [18:49:47] hey Amir1. [18:49:57] Transferring your notes to an etherpad. [18:53:42] Amir1, https://etherpad.wikimedia.org/p/revscoring_spring-cleaning [19:22:56] why I didn't get the notification [19:22:59] okay [19:23:01] cool halfak [19:23:05] thanks [20:30:53] Looks like we have a few bits to merge for revscoring. [20:30:56] I'll get that cleaned up. [21:04:39] halfak: https://phabricator.wikimedia.org/rTESTREVSCORINGAGAINaad8744236c8 [21:04:45] That's amazing [21:05:01] we work in diffusion [21:05:23] Oooh. This does look nice. [21:07:56] So I have been working on a plan for our changes to ORES. [21:08:06] I think that we should do things in this order: [21:08:36] 1. Deploy new models *and* updates to output format described here: https://phabricator.wikimedia.org/T121054 [21:09:20] 2. Analyze the effects of removing protected user features from the prediction models and, if positive, start the conversation about deploying models without those features. [21:09:35] 3. (option) deploy non-protected-user models. [21:09:48] Right now, I'm re-extracting features with protected user features included. [21:10:04] This means that we can advertise *higher fitness* when working on (1) [21:10:21] But once we get to (2), we undoubtedly be looking at *lower fitness* [21:12:55] exactly [21:13:02] that's really good [21:15:13] OK. I think that following through with this plan is going to be my goal for the next couple of weeks. [21:15:29] I'll try to get arabic and urdu wiki set up as well. [21:16:40] * halfak waits for 500k rev_ids to load in his browser [21:17:03] halfak: assign something to me, specially from the etherpad [21:17:45] News response format for ORES. [21:18:12] This will be good. It will give you a light overview of how ORES works in prep for deeper work later. [21:18:55] It's also good that you have been working on the extension because you'll know the pain that our users feel with a change in formats. [21:19:02] https://phabricator.wikimedia.org/T121054? [21:19:19] Yup that's it. [21:19:21] :D [21:19:36] I think our biggest question will be (1) do we support multiple formats (2) if so, do we support it in the get params or completely change our URL structure? [21:21:27] I think it would be good to support multiple formats, like most of APIs [21:22:35] Amir1, generally, I agree. [21:26:55] Woah. urwiki has 99% trusted user edits [21:27:01] I'm guessing that it's all bots [21:27:15] halfak: I also think the transition period is needed [21:27:58] Amir1, why even have a period? We *could* theoretically support the formats indefinitely. [21:28:53] mostly because we add features in the next versions and people should migrate to that [21:29:31] these two outputs are not very different now they are just some simple changes [21:29:42] Amir1, I think that the real concern is adding maintenance load for us. [21:29:45] but later changes would be more drastic [21:30:29] for this one, I'm okay [22:18:04] halfak: I'm trying to make the patch but it seems there is a huge inconsistency here. Compare results of: http://ores.wmflabs.org/scores/enwiki/wp10/4567834/ http://ores.wmflabs.org/scores/enwiki/?models=wp10&revids=4567834 [22:20:00] I use format argument: 1- REST 2- php [22:23:45] Amir1, indeed. different output depending on request structure. [22:28:00] halfak: do you want to keep this inconsistency in v2 too? [22:34:20] Amir1, not convinced. [22:34:41] I could be convinced very easily to make it consistent across different paths. [22:35:15] as a user I always thought this returns the same [22:35:49] also I think we should be more verbose [22:36:13] we should return info regarding the model as well returning about rev ids [22:37:36] halfak: ^ [22:37:53] Amir1, most of the time, a requester won't need that data though. [22:38:05] Seems like it should be optional to me. [22:38:33] Maybe we can have the model, version always come with the rev_ids. [22:38:34] ok [22:38:43] But model fitness statistics would only be available with an option [22:38:50] Since model version is so important for interpreting. [22:39:22] I'm talking different responses [22:39:41] http://ores.wmflabs.org/scores/enwiki/?models=wp10&revids=4567834 returns [22:39:58] {"4567834": {"wp10": {...}}} [22:40:07] Oh yeah. When you have model in the URL, you don't get model back as a key. [22:40:11] but [22:40:13] http://ores.wmflabs.org/scores/enwiki/wp10/4567834/ returns [22:40:29] When you have model in the query string, you could be requesting multiple models and therefor get it back as a key. [22:40:32] {"4567834": {...}}} [22:40:49] same goes for rev id [22:40:54] I agree that this was maybe not the best design decision, but it was made early and we had users when I first considered making it consistent. [22:40:57] but of APIs I've seen (such as mediawiki) returns such reposnses [22:40:59] like query [22:41:23] "such responses"? [22:41:47] I mean try action=query in mediawiki [22:42:02] eventhough you give rev id it responsds with rev id too [22:43:36] halfak: It's okay to keep this in v1 but for v2, do you think we should act this way? [22:43:52] No! :P [22:44:01] great [22:44:06] I've been agreeing with you. [22:44:37] I'm working on it [22:44:55] I see the confusion. By "not convinced", I meant: Not convinced that there's a reason to "keep this inconsistency [...]" [22:45:01] Totally not clear [22:45:23] But I think that within this complex response package, we should favor sending the smallest package we can. [22:46:18] E.g. {"scores": {"enwiki": {"damaging": {"3842834": ...}}}} [22:46:28] That's a deep JSON, but it's actually pretty small. [22:46:31] Just a lot of keys. [22:47:11] OTOH, we could include a bunch of model stuff in every response. [22:48:32] E.g. {"scores": {...}, "model_info": {"enwiki": {"damaging": {"version": "0.0.1", "table": ..., "stats": }))) [22:48:36] etc. [22:49:04] We don't want all that for every response, but we do want a few things. [22:49:08] E.g. a model version. [22:49:16] https://etherpad.wikimedia.org/p/ores_response_structure [22:53:43] OK. I have a set of proposals for response structures. [22:57:41] Amir1, ^ [22:57:58] I was reading the proposal [22:58:06] kk [22:58:06] I made an edit [22:58:24] "model_info" is jargon in revscoring [22:58:38] But we could change that with a refactor. [22:58:45] no [22:58:52] let's stick with info [22:59:26] I think the merged structure is really [22:59:28] god [22:59:33] *good [22:59:52] but the model info should be optional [23:00:02] Yeah. I think you should be able to silence info too. [23:00:14] But we'll have a few things there by default. [23:00:34] okay cool [23:00:34] I think that model.info.version should be there by default [23:00:48] Oh! I have an idea. [23:00:55] I get it [23:01:04] Yeah! [23:01:31] so, {}.scores...version will get you the version of the model that was applied. [23:01:57] {}.scores...scores. will get you a score. [23:02:27] {}.scores...info.stats.roc.auc will get you the ROC-AUC [23:02:41] You'd need to specifically request model_info=stats [23:03:05] or something like that [23:03:12] BRB dog needs fetch [23:03:29] ok [23:03:43] working on it right now [23:20:07] back [23:21:11] Amir1, in the "Deeper merged trees", I took out "info" at the base. What do you think? [23:21:27] It didn't really seem to serve a purpose. [23:21:52] that's really good [23:24:11] Cool.