[02:31:18] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Discovery-Search, 10NewcomerTasks 1.1, 10Growth-Team (Current Sprint): Expose ORES drafttopic data in ElasticSearch via a custom CirrusSearch keyword - https://phabricator.wikimedia.org/T240559 (10Tgr) a:03Tgr [10:56:08] Hi,I was working on this project wikilabels and I am having some problem in starting the server.Can anyone help me? [14:28:27] o/ [14:29:14] I'm going to step away for a bit. We got a bunch of rain & snow here last night so I need to de-ice the sidewalk. [15:05:32] back! [15:06:35] It's bad out there. Very slippery. Hopefully the relative warmth of the day (high of like, 2C/36F) melts some of the ice. [15:32:25] Deploying ORES to beta. [15:39:19] And we ran out of memory. [15:39:20] Oof. [15:43:12] 10ORES, 10Scoring-platform-team (Current): Deploy ORES -- Late Jan 2020 - https://phabricator.wikimedia.org/T243451 (10Halfak) I failed on the deploy to beta. Looks like memory usage is far too high. I'm investigating. [15:53:54] It seems the RSS for just one model is ~400MB. [15:53:59] That's much higher than expected. [15:54:26] We might need to regress back to generating 50 cell vectors. I'd be curious what we can get with reducing our vocab size too. [16:00:54] 10ORES, 10Scoring-platform-team (Current): Deploy ORES -- Late Jan 2020 - https://phabricator.wikimedia.org/T243451 (10Halfak) I'm investigating memory usage. I'm working from a python terminal on my dev laptop. Essentially, I'm tracking VSZ and RSS while running commands. **Before loading anything: ** * V... [16:03:38] 10ORES, 10Scoring-platform-team (Current): Reduce memory footprint of topic models - https://phabricator.wikimedia.org/T243522 (10Halfak) [16:03:41] 10ORES, 10Scoring-platform-team (Current): Reduce memory footprint of topic models - https://phabricator.wikimedia.org/T243522 (10Halfak) >>! In T243451#5827098, @Halfak wrote: > I'm investigating memory usage. I'm working from a python terminal on my dev laptop. Essentially, I'm tracking VSZ and RSS while r... [16:11:03] 10Scoring-platform-team, 10Community comms and outreach, 10CommRel-Specialists-Support (Jan-Mar-2020): Outreach campaign to raise awareness of Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Elitre) [16:12:16] 10Scoring-platform-team, 10Community comms and outreach, 10CommRel-Specialists-Support (Jan-Mar-2020): Outreach campaign to raise awareness of Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Halfak) Hey folks! Sorry to not chime in sooner. We're blocked right now on product support from Aud... [16:17:18] 10Scoring-platform-team, 10Community comms and outreach, 10CommRel-Specialists-Support (Jan-Mar-2020): Outreach campaign to raise awareness of Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Elitre) >>! In T217232#5827161, @Halfak wrote: > Hey folks! Sorry to not chime in sooner. We're bloc... [16:19:49] mornin o/ [16:20:07] Hey accraze! [16:20:31] kevinbazira, glad to hear you were able to get Jade up & running [16:21:14] I think I accidentally introduced a bug into my UI patchset last night, which is why the new proposal workflow didn't work [16:21:28] Our beta deploy failed this morning. Memory usage is much higher than expected and we couldn't deploy to beta. I bet we'd have enough headspace in prod, but I don't like trying it. So I'm working on exploring strategies for minimizing memory usage now. I might restart the pipeline and generate some 50 dimension vectors. [16:22:12] bummer halfak, but good call on not testing in prod :) [16:23:37] * halfak experiments with memory-maps [16:23:54] accraze, any experience with memory-maps that might scare me away from it? [16:26:30] mmaps worked well with KeyedVectors IIRC [16:27:00] Cool. I'm going to give that a try. We need to convert our vectors to gensim pickle objects in order to use memory maps (for some reason) [16:27:04] So that's lame [16:27:23] We'll see how it affects memory usage. [16:31:10] Hmm. The affect seems to be pretty minimal based on what `ps` reports for VSZ and RSS, but the model "load()" method runs almost instantly. So I'm guessing there's something funny going on with how RSS is reported. [16:37:20] 10ORES, 10Scoring-platform-team (Current): Reduce memory footprint of topic models - https://phabricator.wikimedia.org/T243522 (10Halfak) First, I'm trying out memory-maps. I converted out word2vec format into gensim's KV objects with: ` >>> from gensim.models import KeyedVectors >>> model = KeyedVectors.load... [16:47:53] Curses! It looks like you get two files when you save a "*.kv" binary. You also get a ".kv.vectors.npy" file next to it. How annoying. [16:48:34] The bright side is that the filesize is much smaller even when accounting for the two files. [16:48:52] Advice from the internet so far is "get more RAM" >:( [16:50:22] how much extra memory are these new vectors using? [16:52:57] It looks like 400mb per language. [16:53:37] In the past, we used 600 MB for just English. [16:53:53] 400 * 5 langs = 2GB vs. 600MB for one lang. [16:56:49] hey guys will be running late to standup, got some people here doing work on the house, might need to do async if thats ok [16:59:11] Sure. Gotcha. I'll still join the call for a chat (FYI: kevinbazira) [16:59:21] accraze, join late if you can. If not, that's cool :) [17:41:39] halfak the PR LGTM. I've merged it. Good day to you and accraze! [18:32:18] 10Scoring-platform-team, 10Community comms and outreach, 10CommRel-Specialists-Support (Jan-Mar-2020): Outreach campaign to raise awareness of Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Halfak) Indeed. I think a good next step is to get a rep from Audiences (who have promised us some pr... [18:34:06] isaacj, 9.6m words in my English data. [18:34:42] * halfak uses 2500% CPU on stat1007 :D [18:35:09] get on that GPU halfak ;) [18:35:17] though i guess sklearn doesn't support it [18:35:26] and wow, that's a very long tail of vocab [18:37:25] i remember when we were doing vocab coverage analyses with my NYU student researchers with fastText embeddings (can't remember vocab count) but the article from our sample with the least coverage was this: https://en.wikipedia.org/wiki/List_of_names_of_Vishnu [18:46:26] isaacj, lolol that article is so strange. [18:46:35] I'm using fasttext for this. [18:46:41] No GPU support AFAICT [18:47:15] I just wrote a script called word2vec2gensin [18:47:21] *word2vec2gensim [18:47:31] I wonder what we'll convert things to next [18:47:38] word2vec2gensim2classifier [18:48:01] word2vec2gensim2classifier2process2objective2key-result2promotion [18:55:51] +1 the final script. excellent class in modularity :) [19:01:26] Grabbing lunch. [19:06:26] Hi,I was working on this project wikilabels and I am having problem in starting the server.Can anyone help me? [19:25:30] Hey swayamsvk, how are you starting the server? [19:27:37] by running wikilabels dev server it shows "wikilabels:command not found". [19:29:30] try running `./utility dev_server` instead [19:30:29] ^swayamsvk, the directions in the README might be outdated [19:36:05] $ ./utility dev_serverTraceback (most recent call last): File "/media/swayam/A:/gsoc2020new/wiki2/wikilabels/wikilabels/wikilabels.py", line 43, in main package="wikilabels") File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File " importlib._bootstrap>", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 665, in _load_unlocked File "", line 678, in exec_module File " importlib._bootstrap>", line 219, in _call_with_frames_removed File "/media/swayam/A:/gsoc2020new/wiki2/wikilabels/wikilabels/utilities/dev_server.py", line 21, in import docoptModuleNotFoundError: No module named 'docopt'Could not find utility dev_server. [19:40:21] swayamsvk, did you install the package requirements using `pip install -r requirements.txt`? [19:41:56] Nope I was doing only those stuffs which are mentioned in the readme [19:42:53] ahhh I see, yeah we need to update that readme [19:43:36] try installing the requirements and running the server again, that should work [19:50:03] ERROR: Command errored out with exit status 1: command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-aJGfEM/yamlconf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-aJGfEM/yamlconf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', [19:50:04] '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-aJGfEM/yamlconf/pip-egg-info cwd: /tmp/pip-install-aJGfEM/yamlconf/ Complete output (7 lines): Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-aJGfEM/yamlconf/setup.py", [19:50:05] line 28, in install_requires=list(requirements("requirements.txt")), File "/tmp/pip-install-aJGfEM/yamlconf/setup.py", line 14, in requirements for line in open(os.path.join(os.path.dirname(__file__), fname)): IOError: [Errno 2] No such file or directory: '/tmp/pip-install-aJGfEM/yamlconf/requirements.txt' [19:50:05] ----------------------------------------ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. [19:50:12] giving this error [20:00:07] swayamsvk, thats weird, what version of python are you running? wikilabels requires Python 3.x [20:00:56] headin out to an appt & lunch, back in a bit [20:01:02] sorry,need to update that. [20:04:06] o/ swayamsvk I can help with wikilabels too. [20:06:05] still having the same error after updating python. [20:06:52] It's hard to read the error in IRC. Could you use pastebin.com to paste the error? [20:08:33] ok [20:09:50] https://pastebin.com/DN8A6P1W [20:15:46] Is this after running `pip install -r requirements.txt`? [20:16:27] yes [20:17:11] Hmm. Could you try doing "pip install yamlconf"? [20:17:28] This error doesn't seem normal at all. Looks like something failed half-way through installing the requirements. [20:20:50] showing the exact error as the previous one [20:21:17] What version of python are you running? [20:22:44] I just tried a clean install on python 3.5.2 and it worked great. [20:25:58] I think i am having some problem in updating python.First I will fix it and then only I can proceed. [20:26:18] How can I talk to you for future referrence [20:26:19] Gotcha. [20:26:32] This is a great place to ask questions. [20:26:45] ok then [20:57:04] isaacj, Vietnamese has only 1.4M "words" [20:57:10] So that jibes with our expectations. [20:57:38] I'm swinging low with training the next round of models. 50d vectors and vocab of 100k words. [20:58:04] and if they are the equivalent of "syllables" then i expect the long tail is not nearly as large either -- i.e. more uniform usage perhaps [20:58:23] love it -- training should be much quicker too! [20:58:47] Actually seems to take about the same amount of time for the GradientBoosting models. [21:20:17] Woo! We still get a performance boost with the 50d, 100k vocab. I'm training the other models now. The loss we got for going down was not very big. [21:25:13] wikimedia/revscoring#1824 (gensim_vectors - 9654225 : Aaron Halfaker): The build has errored. https://travis-ci.org/wikimedia/revscoring/builds/641096940 [21:27:50] wikimedia/revscoring#1826 (gensim_vectors - 7179e56 : Aaron Halfaker): The build failed. https://travis-ci.org/wikimedia/revscoring/builds/641097545 [21:32:18] mediawiki-utilities/python-mwtext#14 (gensim - 680b6b0 : Aaron Halfaker): The build failed. https://travis-ci.org/mediawiki-utilities/python-mwtext/builds/641100586 [21:32:43] wikimedia/revscoring#1828 (gensim_vectors - 39ca2f7 : Aaron Halfaker): The build was fixed. https://travis-ci.org/wikimedia/revscoring/builds/641099761 [21:41:11] accraze, when you have a minute: https://github.com/wikimedia/revscoring/pull/470 [21:45:45] 10ORES, 10Scoring-platform-team, 10Beta-Cluster-Infrastructure: Failed executing job: ORESFetchScoreJob - https://phabricator.wikimedia.org/T243553 (10MarcoAurelio) [21:48:28] 10ORES, 10Scoring-platform-team, 10Beta-Cluster-Infrastructure: Failed executing job: ORESFetchScoreJob - https://phabricator.wikimedia.org/T243553 (10Halfak) We had a failed deploy earlier today. I'll go clean it up. [21:52:21] 10ORES, 10Scoring-platform-team, 10Beta-Cluster-Infrastructure: Failed executing job: ORESFetchScoreJob - https://phabricator.wikimedia.org/T243553 (10Halfak) Should be back online now. [21:57:33] 10ORES, 10Scoring-platform-team (Current): Reduce memory footprint of topic models - https://phabricator.wikimedia.org/T243522 (10Halfak) Aha! It looks like memory usage is greater when we do not use the `mmap='r'` option. Here's what I see after I run `model = KeyedVectors.load("enwiki-20191201-learned_vect... [22:01:01] 10ORES, 10Scoring-platform-team, 10Beta-Cluster-Infrastructure: Failed executing job: ORESFetchScoreJob - https://phabricator.wikimedia.org/T243553 (10MarcoAurelio) 05Open→03Resolved a:03Halfak I see no further errors in Logstash. Thanks! [22:08:23] mediawiki-utilities/python-mwtext#16 (gensim - 3a6d1ba : Aaron Halfaker): The build was fixed. https://travis-ci.org/mediawiki-utilities/python-mwtext/builds/641114373 [22:10:48] 10ORES, 10Scoring-platform-team (Current): Reduce memory footprint of topic models - https://phabricator.wikimedia.org/T243522 (10Halfak) OK so I've now generated learned vectors for 50c/100k vocab. I just trained the enwiki articletopic model. * The old gnews model: pr_auc (micro=0.718, macro=0.555) * The 1... [22:32:41] merged it halfak [22:32:46] Thanks! [22:35:17] I'll need to update our wheels now for a new deploy. [23:32:14] It's alive! https://github.com/wikimedia/drafttopic/pull/47 [23:32:18] Still waiting on travis. [23:32:25] But I think this'll do it. [23:32:48] All of the models have a minor reduction in fitness in exchange for a major improvement in memory footprint.