[14:37:39] <wikibugs>	 10ORES, 10Scoring-platform-team (Current): Estimate ORES CapEx for FY21 - https://phabricator.wikimedia.org/T249917 (10Halfak)
[17:00:07] <wikibugs>	 10ORES, 10Scoring-platform-team (Current): Estimate ORES CapEx for FY21 - https://phabricator.wikimedia.org/T249917 (10Halfak) Running a test today with uwsgi.  I'm running 96 uwsgi workers on ores-misc-01 to mimic what we have in production.    When running all 96 workers (no celery), we have 30.4GB available...
[17:13:00] <wikibugs>	 10ORES, 10Scoring-platform-team (Current): Estimate ORES CapEx for FY21 - https://phabricator.wikimedia.org/T249917 (10Halfak) Right now, we need 15GB to run all of our models + ~23GB of overhead for buffers for the workers.  Product wants us to host ~15 more models.  15 * 1.2 = 18GB more model memory needed....
[17:18:12] <wikibugs>	 10ORES, 10Scoring-platform-team (Current): Estimate ORES CapEx for FY21 - https://phabricator.wikimedia.org/T249917 (10Halfak) I see two proposals that are interesting here:  1. Let's just buy bigger servers.  If we get servers that have ~128GB of RAM, we'd be able to host a heck of a lot of topic models for p...
[19:28:24] <halfak>	 Async standup time!  Here we go:
[19:28:32] <halfak>	 kevin.bazira:
[19:28:34] <halfak>	 Y:
[19:28:34] <halfak>	 Remove "successfully" from translation messages
[19:28:34] <halfak>	     Based on: https://www.mediawiki.org/wiki/Localisation#Avoid_jargon_and_slang
[19:28:34] <halfak>	 Added MW message key for jade-setpreference without "successfully"
[19:28:34] <halfak>	 Added MW message key for jade-moveendorsement without "successfully"
[19:28:35] <halfak>	 T:
[19:28:37] <halfak>	 I have been working on upgrading MW Vagrant to get MW 1.35.
[19:28:39] <halfak>	 Started looking at the issue about deleting an endorsement that returns an error: jade-bad-content
[19:28:41] <halfak>	     https://phabricator.wikimedia.org/T247973
[19:28:49] <halfak>	 ac.craze:
[19:28:50] <halfak>	 Y: Code review, spent a good amount of time updating MW Vagrant to support RevisionRecord changes for MW 1.35, also made schemas for jade_facet and jade_diff_label
[19:28:50] <halfak>	 T: Continue working on re-enabling hooks and classes to support new schemas and link table structure
[19:28:54] <halfak>	 And me, halfak: 
[19:29:02] <halfak>	 Y: Meetings!  That and working on the uwsgi memory issue.  I made some progress and posted a question on Stack Overflow.  
[19:29:02] <halfak>	 T: I've been working on drawing estimates of our memory usage and how much more memory we need for each model.  The bad news is we need *A TON* more memory to host the topic models (1.2GB each!).  I think that the vectors are not living nicely in shared memory the way we would like them to.