[00:19:04] 10Scoring-platform-team, 10Discovery-Search (Current work), 10Growth-Team (Current Sprint): Newcomer tasks: set initial thresholds for ORES articletopic - https://phabricator.wikimedia.org/T244297 (10Tgr) Hm, I guess putting the threshold config in MediaWiki and applying via the search keyword definition is... [00:49:00] 10Scoring-platform-team, 10Discovery-Search (Current work), 10Growth-Team (Current Sprint): Newcomer tasks: set initial thresholds for ORES articletopic - https://phabricator.wikimedia.org/T244297 (10EBernhardson) > They could also live in the script that loads data from Hadoop to ES (and currently uses a cu... [00:55:58] 10Scoring-platform-team, 10Discovery-Search (Current work), 10Growth-Team (Current Sprint): Newcomer tasks: set initial thresholds for ORES articletopic - https://phabricator.wikimedia.org/T244297 (10EBernhardson) Another concern i just realized with respect to thresholds, will be updating the models. If a n... [03:11:47] 10Scoring-platform-team, 10Discovery-Search (Current work), 10Growth-Team (Current Sprint): Newcomer tasks: set initial thresholds for ORES articletopic - https://phabricator.wikimedia.org/T244297 (10Tgr) >>! In T244297#5858557, @EBernhardson wrote: > I don't fully grok the way thresholds are set, but it see... [06:54:49] 10ORES, 10Scoring-platform-team (Current), 10Operations: Ores celery OOM event in codfw - https://phabricator.wikimedia.org/T242705 (10elukey) It happened again this morning, even if the deployment was rolled back.. from dmesg I can see again celery killed by the OOM. ` [Fri Feb 7 06:33:29 2020] Out of mem... [07:58:53] 10ORES, 10Scoring-platform-team (Current), 10Operations: Ores celery OOM event in codfw - https://phabricator.wikimedia.org/T242705 (10elukey) Simply running `systemctl reload uwsgi-ores` doesn't seem to trigger the issue anymore. On ores1008 there is a difference in impact though: ` Feb 05 06:25:12 ores10... [08:03:33] 10ORES, 10Scoring-platform-team (Current), 10Operations: Ores celery OOM event in codfw - https://phabricator.wikimedia.org/T242705 (10elukey) Might not be related, but https://github.com/unbit/uwsgi/issues/296 seems interesting, pasting here as reference. A lot of people report `--ignore-sigpipe` as useful... [09:08:37] 10ORES, 10Scoring-platform-team (Current), 10Operations: Ores celery OOM event in codfw - https://phabricator.wikimedia.org/T242705 (10elukey) Ok so I did some tests (even with `--ignore-sigpipe`), but nothing good to report. I also straced master and worker processes of uwsgi right after reload, and I can s... [10:41:30] 10ORES, 10Scoring-platform-team (Current), 10Operations: Ores celery OOM event in codfw - https://phabricator.wikimedia.org/T242705 (10akosiaris) Let me add my own finding. Doing `systemctl stop uwsgi-ores` triggers the issue. It's during the stop phase that uwsgi workers go haywire on CPU and memory usage.... [11:06:18] 10ORES, 10Scoring-platform-team, 10Core Platform Team, 10MediaWiki-Special-pages, 10Performance Issue: SpecialRecentChanges::doMainQuery needs tunning - https://phabricator.wikimedia.org/T244569 (10Marostegui) [13:58:33] o/ [13:59:53] (03PS1) 10Halfak: Bumps ORES to HEAD. Address memory issues. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/570893 [15:10:11] (03CR) 10Halfak: [V: 03+2 C: 03+2] Bumps ORES to HEAD. Address memory issues. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/570893 (owner: 10Halfak) [16:26:16] 10ORES, 10Scoring-platform-team, 10MediaWiki-Special-pages, 10Core Platform Team Workboards (Clinic Duty Team), 10Performance Issue: SpecialRecentChanges::doMainQuery needs tunning - https://phabricator.wikimedia.org/T244569 (10Anomie) Why is it deciding it needs to filesort? I see no reason it should ne... [16:35:19] (03PS4) 10Daimona Eaytoy: phpunit: Alter globals via setMwGlobals [extensions/ORES] - 10https://gerrit.wikimedia.org/r/553047 [16:58:30] Confirmed that uwsgi is using only ~450MB per process on beta. [16:58:34] Looks like we're good to go for prod. [17:01:55] 10ORES, 10Scoring-platform-team (Current), 10Operations: Ores celery OOM event in codfw - https://phabricator.wikimedia.org/T242705 (10Halfak) I just deployed a change to beta that dramatically reduced the memory usage of uwsgi processes. See https://github.com/wikimedia/ores/pull/337 This change address... [17:05:21] accraze, coming? [17:38:35] 10Scoring-platform-team, 10Discovery-Search, 10Epic, 10Growth-Team (Current Sprint): [EPIC] Growth: Newcomer tasks 1.1.1 (ORES topics) - https://phabricator.wikimedia.org/T240517 (10MMiller_WMF) [18:03:22] lunch! [18:31:15] (03CR) 10Jforrester: [C: 03+2] phpunit: Alter globals via setMwGlobals [extensions/ORES] - 10https://gerrit.wikimedia.org/r/553047 (owner: 10Daimona Eaytoy) [18:47:56] (03Merged) 10jenkins-bot: phpunit: Alter globals via setMwGlobals [extensions/ORES] - 10https://gerrit.wikimedia.org/r/553047 (owner: 10Daimona Eaytoy) [19:13:05] Hello guys [19:13:13] When i run pip install -r test-requirements.txt [19:13:20] It installs the packages [19:13:38] But on running the tests, I get errors of packages that I didn't install [19:13:58] The packages are not in the test-requirements.txt file [19:15:02] I got the instruction to install test-requirements.txt from the README file [19:15:15] halfak I'll appreciate some clarifiaction on this [19:27:38] Resolved. I had to install the actual requirements.txt file. [19:27:58] I think it should be stated in the README [20:20:57] Hey haksoat! [20:20:59] Was in a meeting. [20:21:05] Reading... [20:21:40] So we've been talking about this a bit. [20:22:27] Hello [20:22:33] There are some commonalities across python packages. The use of requirements.txt is ubiquitous. It would be nice to link to some docs that discuss common python patterns rather than duplicating that everywhere. What do you think? [20:24:05] That will be great [20:24:39] Got any docs in mind? [20:35:41] haksoat, not really. I've been looking around but I haven't found anything that approaches things nicely for a new consumer of a python package. [20:41:15] Found this: https://packaging.python.org/tutorials/installing-packages/#requirements-files [20:52:03] Okay [20:52:08] Taking a look [20:54:56] In the end, it might be good for us to just include the command in the sequence anyway. [21:00:55] What do you mean by sequence in this case? [21:02:42] Looking at that page, I think it'll be fine to just state it in the README. Lots of info on that page. [21:07:42] Also, I'm trying to write tests for the loaded_idioms [21:08:39] But I don't fully understand how the other tests work... eager to have a better understanding of the code base. [21:14:07] I just pushed. Perhaps, I could get some help. [21:34:35] halfak did you send any message while I went offline? [21:35:08] Nope, but now that you pinged, I'll read and respond :) [21:35:45] By 'sequence' I meant in the readme. E.g. we provide a sequence of commands to run in order to set up your environment to run tests. [21:35:54] I'll look at what you pushed [21:35:56] haksoat, ^ [21:36:35] Okay. Yes. I agree with providing a sequence of commands in the README. [21:37:43] Your work in languages/english.py looks good to me. [21:38:06] Would you be willing to submit a PR with some changes to the readme based on what you have learned? [21:38:19] It's nice to have someone new look at it. You see things I take for granted :) [21:40:20] Okay. Great. I'll do that. [21:43:56] Thanks :) [21:57:03] https://www.mediawiki.org/wiki/ORES#Topic_routing [21:57:19] Got some things together. There's still more to do, but I'll need to get back to it later. [21:57:25] I need to hit the road now. [21:57:29] Have a good weekend, folks!