[01:34:43] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Announce deployment of wp10 models to ruwiki community - https://phabricator.wikimedia.org/T138623#3015533 (10Liuxinyu970226) [04:02:00] wiki-ai/revscoring#885 (doc_notes - ba6ea8f : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/200231097 [11:09:18] 10Revision-Scoring-As-A-Service-Backlog, 10Beta-Cluster-Infrastructure, 10ORES: On beta cluster, ORESFetchScoreJob got a HTTP 400 bad request from ores-beta - https://phabricator.wikimedia.org/T157790#3016510 (10hashar) [14:38:58] halfak: I love this graph: https://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Service+Cluster+B+eqiad&h=scb1001.eqiad.wmnet&jr=&js=&event=hide&ts=0&v=527603.88&m=bytes_out&vl=bytes%2Fsec&ti=Bytes+Sent [14:39:31] Starting from Monday, we blocked the bot, so it became normal. You notice there are daily periods. Note that friday is much smaller [14:39:39] because of the minification [14:44:36] Interesting! It'll be interesting to see how that plays out over the next week. [15:27:47] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017153 (10Ladsgroup) Here's the PR: https://github.com/wikimedia/change-propagation/pull/162 And the patch is there too. [15:50:00] Amir1, could you take a moment to review https://github.com/wiki-ai/draftquality/pull/1 ? [15:50:10] I think that repo is ready to enact the review pattern [15:55:05] halfak: hey, sure [15:55:15] Thanks :) [15:55:22] I was in coffee break [15:57:17] * halfak eyes up his empty coffee thermos [15:57:18] :( [15:58:08] I want more coffee, but if I have more coffee, that'll be too much caffeine [15:58:35] o/ these are the questions that try people's souls [15:59:27] This revscoring code is wicked complicated--I should start drinking coffee too. [15:59:38] lol [16:00:12] I got most of the classes into a diagram, fwiw: https://github.com/adamwight/ores-diagrams/blob/master/ores_classes.svg [16:00:13] Hopefully, we can figure out how to make it more simplistic. [16:00:26] * awight simply smiles [16:01:32] btw, I found a use case for non-cartesian product of revs x models [16:01:42] ... in the missing scores step [16:02:48] Oh yeah. Good point. [16:03:26] Not a user use-case. But hidden system use-case. [16:03:30] halfak: I'm wondering what the overrides are for, giving us all kinds of different ways to use a non-standard list of features, caches, etc. [16:04:07] That's easy to preserve, but I don't know that we need such a capability [16:04:28] Oh yeah, so people inject features/datasources for many reasons. [16:04:32] 1. helps with auditing [16:04:43] 2. allows scoring of edits that haven't been saved yet. [16:04:50] Working with arlolra on (2) [16:04:53] nice [16:04:57] He's got a system to filter Tor edits. [16:05:18] The cache injection pattern is a nice generalized way to support all that. [16:05:26] I've used it to study the anon bias before. [16:05:38] I'm thinking that overrides like that should be encapsulated under ScoringRequest [16:06:34] The chunking algorithm should probably interact with ScoringRequest directly. [16:07:33] ah another question: I see that ScoringProcessor forks using both lightweight threads, and process-weight threads [16:07:35] "Chunking"? [16:08:02] sorry--splitting work into parallelizable batches [16:08:45] Oh yeah. So requests to the API are threads (no need for whole processes -- just blocking on IO) and processes to compute features (needs whole processes -- distributes CPU) [16:10:53] (reading https://docs.python.org/dev/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor) [16:11:20] oh. This is about the infamous Global Interpreter Lock. thx [16:12:17] Ahh yeah [16:15:12] I'm straying a bit from the mutex thing I originally planned to do... [16:16:30] At this point, I understand just enough to be dangerous ;-) [16:17:55] What are "injection caches"? Is it just an extra bit of uniqueness to differentiate score cache keys? [16:21:20] injection caches are explicitly from the user. [16:21:34] Whereas the "cache" is often passed around between scoring. [16:22:21] And for the mutex change, it looks like the score and in-progress status are always cached per (rev_id, model_name), never with larger granularity, right? [16:22:43] Hrm I still don't understand what an injection cache would be [16:23:05] awight, link me to a line [16:23:28] awight, (wiki, model, rev_id, version) [16:23:41] that version is the "model version" [16:24:23] e.g. https://github.com/wiki-ai/ores/blob/master/ores/scoring_systems/celery_queue.py#L112 [16:24:39] halfak: ah right, thx there's more to the key [16:25:14] So that "injection_cache" comes from the user via a scoring request. [16:25:40] root_caches are what you get after solving for the base dependencies via the API (IO) [16:26:12] But I meant more that, we save a task's in-progress state per task_id, so I wanted to be sure that each of these tasks is scoring a single (wiki, model_and_version, rev_id). [16:26:23] k following [16:27:11] The root_cache and injection_cache is passed to a scoring job so that features can be extracted using both of them. [16:28:36] By the time you get to https://github.com/wiki-ai/ores/blob/master/ores/scoring_systems/scoring_system.py#L144, the injection_cache no longer serves a purpose. [16:29:00] Ahh! One more thing for caching [16:29:03] And for job tracking [16:29:12] Injection caches themselves are part of the key! [16:29:20] https://github.com/wiki-ai/ores/blob/master/ores/scoring_systems/celery_queue.py#L116 [16:29:45] yes I was seeing that--is the injection cache a class object with a __str__ method? [16:29:53] https://github.com/wiki-ai/ores/blob/master/ores/scoring_context.py#L59 [16:30:07] It's a dict that used to be a JSON blob [16:30:16] Makes sense, although I'm not sure we should be caching injected cache jobs at all [16:30:41] we turn it into a sorted tuple of tuples and then str --> bytes --> sha1 [16:30:46] great [16:30:50] awight, I think that's a fair point. [16:31:06] Right now, we treat them as a legitimate, cache-able thing. [16:31:17] Regardless, we need the unique ID for tracking the jobs. [16:31:20] I guess it has a deterministic effect on the score though, so no harm done [16:31:28] right [16:33:00] So in my (post-mutex) potential refactor, injection_cache would be a part of the ScoringRequestSet, I suppose. [16:33:29] We would iterate over ScoringRequestSet according to an opaque algorithm specific to the requestset object [16:34:10] What abstraction would we get with RequestSet? [16:34:13] e.g. it could be providing items from the cartesian product of [models] x [revs], or in the missing_scores case, it would be a list of [(model,rev)] [16:36:06] Well, the benefit I'm hoping for is that we cut down on internal function complexity, so we have Request, ProcessorState, and Response objects. These can be instance fields so we don't have to pass them around [16:37:38] That would also be the abstraction layer which would be able to report things like "I'm a request with injected caches", while external classes wouldn't need to know about that. [16:38:17] Similarly, a single ScoringRequest would hide details about resolving the revision contents, perhaps [16:38:46] There's still too much I don't understand, but I'll keep refining this line of thought... [16:38:57] I think I see where you're going with this. [16:40:40] E.g. I want a method on ScoreCache that takes a ScoringRequest and a Response and returns a new ScoringRequest and Response with the scores from the cache trimmed from the ScoringRequest and added to the Response. [16:41:12] Mostly running on instinct at this point, but I see lots of complexity around managing batches of data, and coordinated collections, which should be managed by an external class. [16:41:13] Then, for generating the missing scores, I'd so something like ScoringRequest.get_score_groups() [16:41:28] and I'd get (, ) [16:41:52] Sort of... I'd rather not even manipulate requests and responses like that, but have most classes interact with either the singular or the collection abstraction. [16:42:32] Some classes would deal with ScoringRequestSet, when we're piping the work around, and ScoringRequest when we're about to do a specific calculation. [16:48:28] gtg for the day! [16:51:14] o/ [16:55:31] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017473 (10mobrovac) Neither of these is going to get merged until we get enough context to understand the changes and their consequenc... [16:56:09] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017476 (10mobrovac) [17:03:33] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017490 (10Ladsgroup) The difference is very simple. Compare result of https://ores-beta.wmflabs.org/v2/scores/wikidatawiki/?models=dam... [17:05:18] halfak: funnily enough ores in beta is broken too: https://ores-beta.wmflabs.org/v1/scores/wikidatawiki/?models=damaging&revids=421063984 [17:05:43] huh. weird. [17:06:00] Glad it is broken there though -- will be much easier to test. [17:06:42] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3013339 (10Pchelolo) @Ladsgroup Em, there's literally no difference between the responses. They're byte-to-byte identical. [17:06:59] and it's working just in labs [17:11:48] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017537 (10Ladsgroup) Without format=json, response is: ``` { "421063984": { "damaging": { "prediction": false, "prob... [17:16:48] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017545 (10Pchelolo) @Ladsgroup Kk, then a question is whether we want to have pretty JSON at all? Normally in APIs pretty JSON is not... [17:19:43] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017549 (10Halfak) @Pchelolo, we've discussed this and decided that we do. I don't see how this is a changeprop concern [17:20:23] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017554 (10Halfak) > at least I've never seen any APIs that return pretty JSON by default. The mediawiki API returns pretty JSON by de... [17:21:00] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017555 (10Halfak) > Another idea, specific to CP and preaching - you could add support for HEAD requests and return no content at all,... [17:21:49] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017562 (10Ladsgroup) >>! In T157693#3017545, @Pchelolo wrote: > @Ladsgroup Kk, then a question is whether we want to have pretty JSON... [17:39:54] 10Revision-Scoring-As-A-Service-Backlog, 10Beta-Cluster-Infrastructure, 10ORES: On beta cluster, ORESFetchScoreJob got a HTTP 400 bad request from ores-beta - https://phabricator.wikimedia.org/T157790#3017640 (10Ladsgroup) I'm working on this. This is rather weird because we haven't enabled ores in wikidata... [17:50:16] 06Revision-Scoring-As-A-Service, 10ORES: Investigate default JSON minification behavior in production - https://phabricator.wikimedia.org/T157721#3017676 (10Halfak) I've confirmed that: 1. I can't reproduce the behavior with HEAD of wiki-ai/ores by running a simple dev server (via `./utility applications.wsgi`... [17:51:09] 06Revision-Scoring-As-A-Service, 10ORES, 07Performance, 15User-Ladsgroup: Minify json responses - https://phabricator.wikimedia.org/T155931#3017678 (10Halfak) [17:51:11] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3017677 (10Halfak) [18:05:08] 06Revision-Scoring-As-A-Service, 10Deployment-Systems, 10ORES, 06Release-Engineering-Team, and 2 others: Error after "Finished deploy": xrange() arg 3 must not be zero - https://phabricator.wikimedia.org/T157136#3017708 (10thcipriani) p:05Triage>03High [18:10:21] 10Revision-Scoring-As-A-Service-Backlog, 10Beta-Cluster-Infrastructure, 10ORES: On beta cluster, ORESFetchScoreJob got a HTTP 400 bad request from ores-beta - https://phabricator.wikimedia.org/T157790#3017746 (10Ladsgroup) In [[https://github.com/wikimedia/operations-mediawiki-config/blob/938d28a2fc35f868036... [18:29:42] 06Revision-Scoring-As-A-Service, 10ORES: Investigate default JSON minification behavior in production - https://phabricator.wikimedia.org/T157721#3017850 (10Halfak) OK. So here's my hypothesis. We're only setting the Flask config for prettyprint to False. There's no circumstances under which we set it back... [18:29:48] Amir1, https://phabricator.wikimedia.org/T157721#3017850 [18:30:17] The way we need to do this is ridiculous. I really don't want to do this with a config variable. [18:30:21] But I haven't found another way. [18:30:37] I have been reading about JSONEncoder -- trying to work out the right way to do this. [19:00:55] Tried a few things with jsonify. It looks like the that system explicitly ignores you trying to work with it. [19:01:09] So, I've got a cleanup commit on the way that should resolve things. [19:07:14] Amir1, https://github.com/wiki-ai/ores/pull/185 I think this is ready for review. [19:44:52] 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 10Wikilabels: Complete Wikidata item quality campaign - https://phabricator.wikimedia.org/T157495#3018053 (10Glorian_WD) @Ladsgroup : A few days ago @Halfak told me that we need to feed the campaign site with at least 5k Wikidata items to adjust with the... [19:52:32] halfak: I was afk for dinner [19:52:35] I check it ASAP [19:52:38] 06Revision-Scoring-As-A-Service, 10Huggle, 10ORES, 07WorkType-NewFunctionality: Supporting ORES on Huggle - https://phabricator.wikimedia.org/T138060#3018067 (10Halfak) [19:52:42] Thanks [19:54:02] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3018074 (10Halfak) Looks like we already support HEAD. http://flask.pocoo.org/docs/0.12/quickstart/#http-methods [19:55:38] 06Revision-Scoring-As-A-Service, 10ChangeProp, 10ORES, 13Patch-For-Review, and 2 others: Use minified JSON format in ChangeProp - https://phabricator.wikimedia.org/T157693#3018075 (10Pchelolo) Cool! I'll schedule some work on our end to leverage that. [20:13:34] 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 10Wikilabels: Complete Wikidata item quality campaign - https://phabricator.wikimedia.org/T157495#3018151 (10Ladsgroup) It looks okay. [21:39:59] I'm calling it a day [21:40:02] o/ [22:35:19] 06Revision-Scoring-As-A-Service, 10ORES: Investigate default JSON minification behavior in production - https://phabricator.wikimedia.org/T157721#3018470 (10Halfak) It works! Ready for a deploy in Prod http://ores-beta.wmflabs.org/v2/scores/enwiki/ ``` { "scores": { "enwiki": { "damaging": {... [22:36:00] 06Revision-Scoring-As-A-Service, 10ORES: Prod deployment of ORES - https://phabricator.wikimedia.org/T157842#3018471 (10Halfak) [22:36:19] 06Revision-Scoring-As-A-Service, 10ORES: Investigate default JSON minification behavior in production - https://phabricator.wikimedia.org/T157721#3014390 (10Halfak) a:03Halfak [22:36:30] 06Revision-Scoring-As-A-Service, 10ORES: Investigate failed ORES deployment - https://phabricator.wikimedia.org/T157623#3018487 (10Halfak) a:03Halfak [22:36:59] 06Revision-Scoring-As-A-Service, 10ORES: Investigate default JSON minification behavior in production - https://phabricator.wikimedia.org/T157721#3014390 (10Halfak) [22:37:01] 06Revision-Scoring-As-A-Service, 10ORES: Prod deployment of ORES - https://phabricator.wikimedia.org/T157842#3018490 (10Halfak)