[00:00:26] 10Jade, 10Scoring-platform-team, 10User-Testing: Jade Wireframes: Search integration - https://phabricator.wikimedia.org/T212380 (10Harej) [00:08:02] 10Jade, 10Scoring-platform-team, 10User-Testing: Jade Mockups: Search integration - https://phabricator.wikimedia.org/T212372 (10awight) [00:11:02] 10Jade, 10Scoring-platform-team: Jade Mockups: Entry view mode - https://phabricator.wikimedia.org/T212370 (10awight) We already have some code which implements the judgment view mode using a Mustache template, so maybe it would be easier to just tweak this template rather than write a JS layer for prototyping? [00:14:33] 10Jade, 10Scoring-platform-team: Jade Mockups: Entry view mode - https://phabricator.wikimedia.org/T212370 (10Harej) That will depend on what the wireframe ends up looking like; if it's a big radical change, it might be worth just prototyping first before implementing. [00:21:52] 10Jade, 10Scoring-platform-team: Jade Mockups: Entry view mode - https://phabricator.wikimedia.org/T212370 (10awight) >>! In T212370#4836284, @Harej wrote: > if it's a big radical change, it might be worth just prototyping first before implementing. Great, I'm okay either way here. Another potential reason t... [00:24:05] 10Jade, 10Scoring-platform-team: Jade Mockups: Entry view mode - https://phabricator.wikimedia.org/T212370 (10Harej) [00:24:10] 10Jade, 10Scoring-platform-team (Current), 10Design, 10Patch-For-Review: Jade Implementation: Entry view mode - https://phabricator.wikimedia.org/T208819 (10Harej) [00:24:57] 10Jade, 10Scoring-platform-team, 10User-Testing: Jade Mockups: Watchlist integration - https://phabricator.wikimedia.org/T210560 (10Harej) [00:25:00] 10Jade, 10Scoring-platform-team (Current), 10Design: Jade Implementation: Watchlist integration - https://phabricator.wikimedia.org/T201361 (10Harej) [00:25:09] 10ORES, 10Scoring-platform-team (Current), 10Analytics: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) a:03awight [00:26:26] 10Jade: Jade Implementation: Entry edit mode - https://phabricator.wikimedia.org/T212385 (10Harej) p:05Triage→03Normal [00:26:39] 10Jade: Jade Implementation: Entry edit mode - https://phabricator.wikimedia.org/T212385 (10Harej) [00:26:42] 10Jade, 10Scoring-platform-team, 10Design: Jade Mockups: Entry edit mode - https://phabricator.wikimedia.org/T199128 (10Harej) [00:28:02] 10Jade, 10Scoring-platform-team, 10drafttopic-modeling: Discuss whether we can deploy JADE for draft topic without fixed categories - https://phabricator.wikimedia.org/T203032 (10Harej) [00:28:11] 10Jade, 10Scoring-platform-team (Current), 10Operations, 10TechCom, and 3 others: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381 (10Harej) [00:31:05] 10Jade, 10Scoring-platform-team (Current), 10Operations, 10TechCom, and 3 others: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381 (10Harej) [00:31:09] 10Jade, 10Scoring-platform-team (Current), 10Design, 10Patch-For-Review: Jade Implementation: Entry view mode - https://phabricator.wikimedia.org/T208819 (10Harej) [00:33:43] 10Jade: Jade Implementation: Diff view controls - https://phabricator.wikimedia.org/T212387 (10Harej) p:05Triage→03Normal [00:34:00] 10Jade, 10Scoring-platform-team, 10User-Testing: Jade Mockups: Diff view controls - https://phabricator.wikimedia.org/T210558 (10Harej) [00:34:03] 10Jade: Jade Implementation: Diff view controls - https://phabricator.wikimedia.org/T212387 (10Harej) [00:37:47] 10Jade: Jade Implementation: Search integration - https://phabricator.wikimedia.org/T212388 (10Harej) p:05Triage→03Normal [00:38:07] 10Jade, 10Scoring-platform-team, 10User-Testing: Jade Mockups: Search integration - https://phabricator.wikimedia.org/T212372 (10Harej) [00:38:09] 10Jade: Jade Implementation: Search integration - https://phabricator.wikimedia.org/T212388 (10Harej) [12:04:02] o/ [12:09:54] 10ORES, 10Scoring-platform-team, 10RESTBase, 10RESTBase-API, and 2 others: Use RESTBase for ORES precaching - https://phabricator.wikimedia.org/T166161 (10mobrovac) [12:18:30] 10ORES, 10Scoring-platform-team, 10RESTBase, 10RESTBase-API, and 2 others: Set up revscoring entry points in RESTBase - https://phabricator.wikimedia.org/T107196 (10mobrovac) [12:27:23] 10ORES, 10Scoring-platform-team (Current), 10Analytics: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10JAllemandou) The query you bookmarked is doing what you expect I guess :) The next steps are to create the associated table and create new partitions based on the query.... [12:54:24] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Analytics, 10Core Platform Team Backlog (Designing), 10Services (designing): ORES hook integration with EventBus - https://phabricator.wikimedia.org/T201869 (10mobrovac) [12:55:34] 10ORES, 10Scoring-platform-team, 10Core Platform Team Backlog (Designing), 10Services (designing): Merge ORES precaching with ORESFetchScoreJob - https://phabricator.wikimedia.org/T201868 (10mobrovac) [13:18:26] (03PS1) 10Ladsgroup: Bump editquality to HEAD [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/480959 [13:18:43] https://gerrit.wikimedia.org/r/#/c/mediawiki/services/ores/deploy/+/480959 [13:18:52] I'm going to leave here for a while then I go beta [13:32:09] akosiaris: hey, after deploying this ^ it will increase the memory footprint, I think we might end up needing to reduce the number of celery workers [13:32:30] I really like to find a better long term solution for the memory footprint issue [13:32:43] by how much ? [13:32:57] or will beta tell us ? [13:33:10] I can estimate [13:34:15] it increases the size by 10MB * number of workers [13:34:57] so about 1G [13:35:11] we 'll survive from what I see [13:35:12] yup [13:35:25] Fingers crossed [13:35:53] but it brings another long-term issue. How we can add support for new wikis without ending up having such issues [13:41:27] Amir1: the only way I can think of is by implementing some kind of lazy loading and GC in models loading [13:41:39] aka, don't load elwiki until a request for elwiki shows up [13:42:14] and then if no request for elwiki has arrived for X seconds unload the model [13:42:29] but this come into directly contradiction with the precaching architecture [13:42:52] where it's guaranteed we will see requests for elwiki even if noone is using rc at that point in time [13:43:41] loading a model into memory takes lots of time [13:43:50] the other way would be to shard the wikis. Doing routing to a specific set of workers per wiki [13:44:04] that's why restarting services takes so much time [13:44:05] what we currently do for mysql basically [13:44:13] yeah, I've been thinking about it a lot [13:44:20] but that is going to end up being very cumbersome as well [13:44:27] there's a lot of logic in mediawiki for this [13:44:29] yes, I even think we should just map to mysql shards [13:44:44] so don't end up doing all the maintaince [13:45:07] the compartmentalization is also rather restrictive [13:45:34] we end up doing weird maintenances that can only be done during switchovers in order to shuffle around the wikis based on usage [13:45:49] but this probably does not apply so much here [13:46:23] Amir1: btw, would it make sense to move the models somewhere else and not in git repos? [13:46:54] I 've been thinking that celery worker ORES docker images are probably gonna be gigantic just due to the models [13:46:59] and thinking of ways to solve it [13:47:07] akosiaris: I'm for it but halfak disagrees because we can access to historical models [13:47:14] we have git lfs to reduce issues [13:47:51] akosiaris: yeah, can we have different docker files/pods for each wiki? [13:48:04] oh maybe I gave you the wrong impression. I don't mean models should not be in git. Just wondering if we can avoid deploying them with the code [13:48:27] it's data after all, maybe we should be loading them instead and not deploying them ? [13:49:11] I know booking.com loads them from hadoop on every worker start [13:49:23] it'll be complex because the git repository containing models actually has lots of code too that are needed [13:49:31] but it's doable [13:49:56] different docker files/pods per wiki ? hmmm [13:49:59] we can put them on archiva I guess :/ [13:50:21] or if it's hard, different docker files/pod per shard [13:50:26] I have this feeling someone will pretty soon show up (if they haven't already) with some AI models database [13:51:37] heh https://mldb.ai/ [13:51:54] someone is at least already trying. No idea if it's any good [13:52:56] Amir1: depends I guess on the loading models paradigm. If they are still going to be part of the repos it's probably not easily doable. If it would be possible to split code and data, then yeah [13:53:23] all it would take is setting e.g. an ENV var saying which models the worker should load [13:54:36] well, even if the code + models are going to be deployed together it's essentially doable [13:54:52] just telling the worker which models it should load [13:55:11] or "all" in order to have a few workers as fallback [13:55:56] that should keep the memory usage down and allow to spawn workers based on the usage [13:56:45] but it is still not very easy to inform the process of how many of each it should spawn [13:58:01] we can do some changes to make it doable via ENV [13:58:03] I like that [13:59:31] (spawning the contexts/models I mean) [14:20:10] yeah but keep in mind this adds a need for configuration management [14:20:33] simply keeping track of which wikis are loaded and based on what criteria can be difficult [14:21:06] especially because solving it means making a number of assumptions [14:21:35] like "how many hosts? how many workers/host? how many workers then per shard?" and so on [14:27:02] yeah, I know :/ It's just I can think of anything better. [14:27:32] akosiaris: one thing is that celery basically copy paste all models in each process [14:27:50] so in general it's not that big but lots of workers copy paste it [14:28:15] I was thinking of sharing them somehow but can't think of anything good [14:28:53] Maybe different type of concurrency like eventlet http://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html [14:29:02] I haven't tested any of them though [14:29:41] ah that reminds me [14:29:51] btw all these models are being instantiated only once [14:29:58] and the memory is being shared across all workers [14:30:18] that's a linux copy on write mechanism and it's the reason this is working as it is currently [14:30:37] if you look at the supposed RSS of all these process the sum is way way larger than the actually usage [14:30:39] oh I thought it's celery [14:30:57] yeah it's basically 200% [14:31:08] if you moved the model loading code in a place that the loading happens after the workers are instantiated you lose the benefit [14:32:40] I see [14:33:11] don't fret too much, that benefit is going to probably go away in kubernetes land as well [14:34:31] I know :D [14:34:51] that's why I was thinking of sharding [14:55:55] wikimedia/editquality#430 (revscoring_2.3.0 - b167716 : Aaron Halfaker): The build passed. https://travis-ci.org/wikimedia/editquality/builds/470547942 [15:30:18] All editquality models rebuilt. [15:30:29] articlequality and draftquality need babysitting. [16:05:59] o/ Amir1 [16:06:06] Added some notes to https://github.com/wikimedia/ores/pull/306 [16:06:07] hey [16:06:14] Already answered all [16:06:31] I think we should pull redis-service out of the unit tests. You should be able to construct the Redis score_cache with a mock Redis connection. [16:07:02] That's a lot of work [16:07:08] for little gain [16:07:15] Is it? It seems like mocking two function calls is pretty straightforward [16:07:19] It's standard practice for a reason. [16:07:30] Two function calls: get/set [16:08:20] https://github.com/wikimedia/ores/blob/a80bdfa13bc79a8c68be326379a12f254fa9360e/ores/score_caches/redis.py#L30 [16:08:31] https://github.com/wikimedia/ores/blob/a80bdfa13bc79a8c68be326379a12f254fa9360e/ores/score_caches/redis.py#L43 [16:08:33] *setex [16:08:34] wikimedia/ores#1228 (celery_tests - 03208cd : Amir Sarabadani): The build has errored. https://travis-ci.org/wikimedia/ores/builds/470578788 [16:09:18] the thing is in any case that redis is not available you can just skip all redis by running pytest -m "not redis" [16:13:19] + how you are going to mock redis in celery tests? I'm writing tests for that too [16:20:44] In all cases, redis just needs get and setex mocked. They are simple key-value set and get patterns so it should be really easy. [16:21:27] I'd say that redis is possibly the easiest thing to mock that I have ever come across. [16:21:30] :P [16:23:36] yes but how I'm going to inject the mock into celery app? [16:23:53] apps? [16:24:12] Oh I see what you mean. I think celery will need to be mocked. [16:24:41] http://docs.celeryproject.org/en/latest/userguide/testing.html [16:24:49] "To test task behavior in unit tests the preferred method is mocking." [16:26:12] I think you can just run celery in single thread mode too. [16:26:47] This seems relevant: http://docs.celeryproject.org/en/3.1/configuration.html#celery-always-eager [16:52:54] o/ awight! [16:52:57] Good morning :) [16:53:29] I think Amir1 and I could use a third opinion on unit testing strategy if you have time for it. [16:53:38] I'm just about to jump into a marathon goals meeting. [16:53:46] I'll send you some scrollback. [16:54:32] Scrollback: https://phabricator.wikimedia.org/P7936 [16:58:02] heya [16:58:11] hrm my highlighting is borken [16:58:43] +1 that the external service should be mocked [16:59:45] probably we should also mock celery like you're saying, for unit tests. [17:00:03] * halfak --> 3 hour marathon meeting. [17:00:06] yup [17:00:09] sorry to hear [17:00:45] BTW, I'm almost done rebuilding all of the models. :) I should have a full set of PRs later today. Probably not going to make the deployment cutoff but we can get it onto beta and labs :D [17:03:20] IIRC we had decided not to deploy today cos since so many people, including RelEng, are out tomorrow, so this is effectively a Friday. [17:03:28] +1 beta and labs though [17:11:35] I'm curious about why all models had to be rebuilt, is there an sklearn upgrade? [17:24:28] 10Jade, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) @daniel or anyone else from TechCom want to help close this out? Code review is on hold pending DBA... [17:31:31] 10Jade, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Marostegui) I cannot really provide more feedback on the code itself apart from what I commented about the q... [17:37:02] 10Jade, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) Moving to the RFC inbox, so TechCom will look at it during the next meeting. Since DBA have approved... [17:46:54] 10Jade, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4838242, @Marostegui wrote: > I cannot really provide more feedback on the code itsel... [17:52:57] 10Jade, 10Scoring-platform-team, 10DBA: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10awight) [17:53:15] 10Jade, 10Scoring-platform-team, 10DBA: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10awight) [17:53:21] 10Jade, 10Scoring-platform-team (Current), 10Operations, 10TechCom, and 4 others: Deploy JADE extension to production - https://phabricator.wikimedia.org/T183381 (10awight) [17:53:39] HareJ: :) just filed the first post-deployment task for Jade. [17:55:22] Hmm, what makes it a post deployment task? [17:56:00] ah, cos it relies on real-world data like 1M rows [18:12:06] Ah, the first explicitly post deployment task, and not “post deployment because we’re deprioritizing it” [18:12:18] :) haha that's it, what I meant [18:19:27] 10Jade, 10Scoring-platform-team, 10DBA: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10Marostegui) p:05Triage→03Normal [19:24:01] 10ORES, 10Scoring-platform-team (Current), 10Analytics, 10Patch-For-Review: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) With the schema above, I was able to insert rows using this query: ` lang=sql set hive.exec.dynamic.partition.mode = 'nonstrict'; insert i... [19:26:04] 10Jade, 10Scoring-platform-team, 10DBA: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10awight) [19:39:56] 10ORES, 10Scoring-platform-team (Current), 10Analytics, 10Patch-For-Review: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) @Halfak In CR, we were discussing the prediction vs. probability fields. I'm currently planning to only include the probabilities, since t... [19:54:53] 10ORES, 10Scoring-platform-team (Current), 10Analytics, 10Patch-For-Review: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Halfak) I think the "attractive nuisance" of the "prediction" field is debatable for editquality and probably not applicable to other models. It f... [19:59:21] Massive meeting complete [19:59:32] awight, I'm around if you want to talk more about the "prediction" field. [19:59:40] I do see what you are saying about it. [20:01:51] halfak: Thanks for the response. I'm okay with including and yr explanation makes sense to me. [20:02:22] OK. I think we should keep this conversation alive though. [20:02:31] Maybe some models shouldn't make a "prediction" [20:02:49] I was thinking about the drafttopic model where the prediction is too specific and not sensitive enough. [20:03:00] On the other hand, we might be able to tune that a bit so that it is more useful. [20:07:36] Or we could even include predictions made with configured cutoffs, basically push the RCFilter thresholds into ORES configuration... [20:08:09] halfak: how is our goal setting going? [20:08:17] Just finished the marathon meeting. [20:08:27] No pushback on our limited goals with one exception. [20:08:46] VC wants us to capture *everything we do* in a goal (which doesn't quite make sense, but OK) [20:09:04] So I put a catchall for building models and doing research projects on demand [20:09:38] baahaha [20:09:52] don't forget the timecard column for filling out timecards [20:09:59] literally. bureaucracy is not cheap. [20:14:42] PROBLEM - puppet on ORES-redis02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:17:26] There were no complaints about us filing "annual planning/strategy" as a goal [20:17:46] Isn't our nonprofit rating damaged by this... ugh [20:31:11] I was wondering if this would be considered non-programmatic. [20:31:14] I'm not sure. [20:33:19] * halfak --> lunch [20:35:00] 10ORES, 10Scoring-platform-team (Current), 10Analytics, 10Patch-For-Review: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) [20:38:41] Noticed something odd about the event.mediawiki.revision_scored schema, it has prediction as an array of string. I'm assuming that's only to have empty set when the scoring errored? [20:42:42] RECOVERY - puppet on ORES-redis02.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:49:32] 10ORES, 10Scoring-platform-team (Current), 10Analytics, 10Patch-For-Review: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10awight) @JAllemandou I need another clue here, this query works when I run each insert separately but when run together I get an error about 'No pa... [20:53:27] 10ORES, 10Scoring-platform-team (Current), 10Analytics, 10Patch-For-Review: Wire ORES scoring events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Nuria) @awight maybe moving to IRC for this might help, i think your query is missing a ";" [21:10:28] o/ might be a while, gotta run! [21:46:55] o/ [21:47:02] Almost done rebuilding all of the models. [21:47:14] Looks like maybe we can go to beta tomorrow. I hope to get everything lined up. [22:15:04] 10Jade, 10Scoring-platform-team, 10Gerrit: Clone gerrit repo mediawiki/extensions/JADE to mediawiki/extensions/Jade - https://phabricator.wikimedia.org/T212180 (10MarcoAurelio) According to https://gerrit.wikimedia.org/r/Documentation/intro-project-owner.html#project-rename Gerrit core does not support the r... [22:17:21] 10Jade, 10Scoring-platform-team, 10Gerrit: Clone gerrit repo mediawiki/extensions/JADE to mediawiki/extensions/Jade - https://phabricator.wikimedia.org/T212180 (10Paladox) We should create a new repo for now, since the rename plugin does not support NoteDB yet. [23:53:44] Just finishing up the last article quality models. Ran into some weirdness with grep of all things. [23:54:08] Grep does not understand unicode characters from the command line. You must provide an escape sequence. This is new(ish). [23:54:32] I only noticed because I needed to rebuild the dataset from XML dumps and thus re-sample.