[09:41:53] akosiaris: hey, I just saw you merged those patches. Awesome! thanks :) [09:42:22] o/ [09:42:54] o/ [09:43:21] I think this is the next step but I might be wrong https://gerrit.wikimedia.org/r/#/c/278990/2 [09:44:39] I think we need to wrok out the sudo rulels too [10:59:00] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Setup varnish endpoint for ORES - https://phabricator.wikimedia.org/T124203#2344431 (10Ladsgroup) With lots of talks with @Halfak, we finally came to an agreement that we want https://ores.wikimedia.org [11:54:17] akosiaris: YuviPanda, halfak and I talked last night about using scap in our labs setup. https://phabricator.wikimedia.org/T136488#2342158 Yuvi wrote some notes about it in here. Please check [12:05:59] ok, will do [12:21:06] thanks :) [13:48:55] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2344730 (10Ladsgroup) 05Open>03Resolved [14:05:30] 06Revision-Scoring-As-A-Service, 10Wikilabels: [Investigate] Intermittent performance issues with wikilabels - https://phabricator.wikimedia.org/T130872#2149118 (10Ladsgroup) I've been monitoring time to generate requests in the past few days. It works perfectly fine. [14:12:47] Hey Amir1, do you know if the models are unique per context? Or do different contexts sometimes share the same model? [14:19:47] o/ [14:20:01] Hey schana. [14:20:05] Is now a good time to talk about ORES refactor and the in-progress-task issue? [14:20:09] Hi halfak [14:20:10] sure [14:20:17] * halfak is thinking that we should build some really simple tests to explore the complication of a few potential solutions. [14:20:45] * schana grabs headphones [14:20:45] I think we can formalize the problem in a really simple way. [14:23:16] halfak: were you wanting to chat over hangouts? [14:23:22] or just on IRC [14:23:59] Hmm... IRC I think. [14:24:03] Got a meeting coming up shortly [14:24:05] okay [14:25:44] Our two methods: https://gist.github.com/halfak/73ca4341e0181d3b4a9b55127690ca60 [14:26:08] that "random.lognormvariate(-0.69, 1)" will produce a representative score timing. [14:27:51] For https://etherpad.wikimedia.org/p/ores_task_tracking [14:30:28] is network bandwidth a concern between worker nodes? [14:32:30] Yes. [14:33:08] what's using it up currently? [14:34:06] "using it up"? [14:34:50] the network bandwidth - I'm concerned with the 'single celery task' per set of requests [14:35:30] as opposed to passing data between different tasks [14:35:35] (like a set of features) [14:36:38] I think it's fine as a logical task, but limiting it to a single task in celery seems premature [14:40:42] halfak: schana I was afk [14:40:43] hey [14:43:28] hi Amir1 [14:43:59] name of models are the same so we have damaging for fawiki and enwiki [14:44:07] but they are not the same [14:44:14] is that true always? [14:45:51] AFAIK yes [14:45:53] halfak: for complex requests (multiple models, revisions), would overriding features be available on an individual basis, or would it be global for the entire request? [14:46:12] sorry... 5% here while in meeting :) [14:46:23] no worries :D [14:46:24] https://github.com/wiki-ai/editquality/tree/master/models [14:46:39] schana, the amount of data shared between model applications is quite big. [14:47:01] schana: these are model files, none of them are the same [14:47:10] thanks Amir1 [14:50:23] yw [14:53:48] Generally, we don't transfer model files between tasks, the task worker will have the models in memory. [14:54:18] The big data transfer issues will be around "dependency_caches" generate while extracting features. [15:07:45] * halfak is back to 100% [15:08:35] looking at schana's question "would overriding features be available on an individual basis, or would it be global for the entire request?" [15:08:40] I'm not sure what you mean here. [15:08:59] it is currently possible to override features, right? [15:09:11] Oh yes. [15:09:18] with the more complex requests that have multiple models and revisions, how do you want the behavior? [15:09:24] It is per revision, but global to all models applied to revision. [15:09:49] E.g. {rev_id: {injection_cache}, rev_id: {injection_cache}} [15:10:22] how is it included in the request? [15:11:07] ?cache=...some json... [15:11:16] (Usually, you'd do it as a POST var) [15:11:36] When you are scoring a single revision, you can use feature names as query params [15:11:43] * schana grumbles about the meaning of 'RESTful' [15:11:50] okay :) [15:12:09] schana, not sure what the concerns are about RESTful here [15:12:17] JSON is just an encoding [15:12:59] I was talking about it being a POST var [15:13:33] Oh. Sure. But having a payload that big in a query param can cause its own problems. [15:13:45] Technical limitations --> Allow POST [15:13:56] E.g. you can inject the full text of the article. [15:14:01] That gets really big really fast [15:14:10] the grumbles were more about shoehorning REST into http [15:14:16] ?datasource.revision.text=... [15:20:38] :) [15:23:15] Amir1: halfak a quick q: I see a [submodule "wb-vandalism"] in .gitmodules of ores-wikimedia-config repo but git submodule status does not list it. Any idea why ? [15:23:28] or alternatively, do we need it ? [15:23:36] akosiaris: because it got deprecated and merged into revscoring [15:23:38] akosiaris, used to exist. We don't need it anymore. [15:23:50] hey halfak :) [15:23:53] o/ Amir1 [15:23:57] ah, great [15:23:58] thanks! [15:24:10] awesome [15:25:56] halfak: right now I'm working on adding the precaching into end of ORES extension requests [15:26:14] Awesome. [15:26:32] had some issues with connecting with gerrit.wikimedia.org (it was trying to use my prod key, i need to update docs in wikitech) [15:26:55] As you may see in the history, schana and I are looking into the task ID problem I described in the meeting yesterday. [15:29:07] yeah, I'm pretty happy about it :) [15:29:25] I have nothing to add. I feel so stupid about it :D [15:29:59] :P Thinking about hard thing is the best part of software engineering work. [15:30:05] Fixing old problems is boring IMO. [15:30:08] Necessary though [15:33:42] :D [15:41:28] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-articlequality : [Explore] Spam and Vandalism new page creation - https://phabricator.wikimedia.org/T135644#2345041 (10Halfak) a:03Soni [15:41:39] 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : [Explore] Spam and Vandalism new page creation - https://phabricator.wikimedia.org/T135644#2305525 (10Halfak) [15:42:53] SoniWP just picked up https://phabricator.wikimedia.org/T135644 [15:42:57] Welcome to the team :) [15:43:08] He's got some updates on preliminary data for modeling here. [15:43:19] And we have fresh problems for how to *get* the data! [15:48:51] Is there a preferred way to share images, halfak? [16:00:57] schana, if they will be re-used at all, commons. [16:01:05] Otherwise, I just use imgur [16:03:05] (03PS1) 10: Add extra_param to pass to the ORES service and use precaching [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [16:04:11] (03CR) 10: "Untested" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [16:05:59] Here's what I'm playing around with at the moment https://usercontent.irccloud-cdn.com/file/h0sGVMVP/ORES_in_progress.png [16:07:30] Having the extracted features live in some sort of temporary data store makes things easier IMO [16:13:11] (03CR) 10Krinkle: Add extra_param to pass to the ORES service and use precaching (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [16:13:12] schana, not sure I see what you are thinking re. async feature extraction. [16:15:10] (03PS2) 10: Add extra_param to pass to the ORES service and use precaching [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [16:15:17] It felt not to fit with the rest, and I didn't see any reason why it would have to live with everything else [16:15:39] It could be "2.5", if that makes more sense [16:15:46] (03CR) 10: "Thanks Krinkle, done" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [16:16:36] schana, just wondering about "async", not the divide [16:16:51] It almost seems as if it could be it's own system [16:16:56] *its [16:17:14] Features are model-specific [16:17:27] But I guess I can see what you mean. [16:17:37] In feature extraction, you'll get both IO and CPU. [16:18:00] IO while base datasources are fetch'd and CPU while features are extracted from base datasources. [16:18:19] In model application, you'll get trivial CPU and no meaningful IO. [16:18:46] We batch some of that IO in multi-rev_id requests [16:19:40] My thought was that there would be a foo that builds up a list of features that need extracted for a given revision, and then passes that list somewhere else to be extracted [16:20:00] Ahh yeah. I hear you. [16:20:58] then the individual models are applied using the (temporary) cache of features [16:21:07] https://github.com/wiki-ai/ores/pull/144/files#diff-65e9fed65dcc67b10f241f7e4c7eedf7R95 [16:21:19] This generates the base datasources for fetching from the API [16:21:41] This actually does the work of getting them extracted: https://github.com/wiki-ai/ores/pull/144/files#diff-65e9fed65dcc67b10f241f7e4c7eedf7R101 [16:22:29] This then does the CPU for solving the features based on the cache of fetch'd dependencies: https://github.com/wiki-ai/ores/pull/144/files#diff-65e9fed65dcc67b10f241f7e4c7eedf7R61 [16:22:40] So IO and CPU are fully separate. [16:25:01] I suppose the process_score method includes the little bit of CPU from actually applying the model. https://github.com/wiki-ai/ores/pull/144/files#diff-65e9fed65dcc67b10f241f7e4c7eedf7R75 [16:25:07] So that's really all the CPU work [16:25:19] Note how "solve_features" is called within [16:26:23] akosiaris: thanks! Is it ready to deploy? [16:26:34] * Amir1 does a little dance [16:27:00] So, coming back to this, it seems to me that we have nothing to do async while the datasources are being fetched. But that we can split up the CPU work of solving features and applying the model. This is currently how we operate. [16:27:00] Amir1: I am currently running puppet on the nodes [16:27:06] I 'll let you know [16:27:09] Running to lunch. be back in a bit. [16:27:26] awesome [16:40:17] (03PS3) 10: Add extra_param to pass to the ORES service and use precaching [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [17:16:05] Amir1: so, redis password. Seems like the last in a roadblock of obstacles I 've had to go through [17:16:27] I see BROKER_URL supports it, I 'll amend the config for that [17:16:45] so score_processors seems to be fine [17:16:57] but what about scores_caches ? how do I pass a redis password there ? [17:17:16] akosiaris: let me check [17:19:05] akosiaris: you can use the config.pp [17:19:29] yeah, I mean what attribute should I pass ? [17:19:43] is it so simple as 'password' => 'mypass' ? [17:19:47] or something else ? [17:20:04] akosiaris: it turns params to yaml configs [17:20:17] I know that [17:20:26] Back! [17:20:33] which param is the password is my question [17:20:45] Got in a conversation about epistemology and Truth [17:20:54] * halfak likes visiting the University [17:21:00] hoping that is it supported ofc [17:21:48] okay [17:21:48] we support it [17:21:48] let me check it [17:23:31] halfak: do we have a password for redis supported [17:23:31] ? [17:23:37] I'm trying to find it [17:23:52] We can specify it in the config [17:23:56] for ores specifically, not for celery [17:24:06] OK. Checking it out. Sec. [17:24:07] for celery, it's supported due to the BROKER_URL syntax [17:26:01] https://gist.github.com/halfak/51a537d5342b3379162f0848dfb47ec7 [17:26:05] This is how I would do it. [17:26:11] Note the file is named 55-redis.yaml [17:26:27] halfak: ok that looks cool, thanks [17:26:39] :D [17:26:57] thanks [17:27:45] I need to be afk for ten min [17:29:14] (03CR) 10Krinkle: Add extra_param to pass to the ORES service and use precaching (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [17:42:30] akosiaris: thoughts on https://gerrit.wikimedia.org/r/#/c/291967/ [17:46:02] (when you have time / are done deploying, that is) [17:47:12] YuviPanda: yeah, will take a while I think, but I 've seen it, as well as the discussion. I am aware [17:47:19] ok [17:49:44] back [18:01:39] (03CR) 10: Add extra_param to pass to the ORES service and use precaching (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [18:02:06] (03PS4) 10: Add extra_param to pass to the ORES service and use precaching [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [18:02:23] Amir1: halfak jfyi, we're performing maintenance on graphite.wmflabs.org over the next few days, so you might see intermittent outages and data loss on graphite [18:03:26] * Amir1 loves a better graphite [18:08:15] halfak: does "https://ores.wmflabs.org/scores/testwiki/?models=damaging&revids=90&precache=1" looks okay? (I'm talking about the precaching param) is it being sent correctly? [18:19:30] (03CR) 10: [C: 031] "Tested, works like a charm" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [18:29:50] https://phabricator.wikimedia.org/T136721 [18:29:58] gerrit-wm hates me [18:36:19] halfak: ^ [18:37:41] Weird [18:38:43] halfak: beside that I mean the upper one :D [18:39:10] "precache=true" [19:01:03] (03PS5) 10: Add extra_param to pass to the ORES service and use precaching [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292163 [19:34:37] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Setup LVS for ORES - https://phabricator.wikimedia.org/T124202#2346077 (10Ladsgroup) We need this merged too: https://gerrit.wikimedia.org/r/#/c/291945/ [20:40:36] halfak: around? [21:30:26] o/ Amir1 [21:30:30] sorry just packing up to get on bike [21:30:33] but here for 2 min [21:30:50] nothing much [21:30:59] halfak: enjoy the bike [21:31:49] BTW, I have working code here: https://github.com/halfak/dependent_task_testing [21:31:55] Write-up coming soon [21:33:41] good luck :) [23:29:49] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review, 07Puppet: ORES-staging is broken due to service::uwsgi mandatory scap::target invoke - https://phabricator.wikimedia.org/T136488#2347636 (10Ladsgroup) a:05Ladsgroup>03None [23:30:30] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review, 07Puppet: ORES-staging is broken due to service::uwsgi mandatory scap::target invoke - https://phabricator.wikimedia.org/T136488#2336803 (10Ladsgroup) Removing myself an the assignee since yuvi is working on it [23:31:02] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review, 07Puppet: ORES-staging is broken due to service::uwsgi mandatory scap::target invoke - https://phabricator.wikimedia.org/T136488#2347651 (10yuvipanda) a:03yuvipanda