[00:34:24] <wikibugs>	 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10CDanis) Around 23:48 we got another user report in #wikimedia-ai of Redis exceptions when using the ORES API.
[06:23:30] <wikibugs>	 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Joe) The request hammering from okapi is continuing, we might need to ban the UA at the edge.    @RBrounley_WMF can you ensure that the reque...
[07:17:07] <wikibugs>	 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Joe) I've created the patch above, that can be used in case the okapi causes further problems to ORES. It's here as a stopgap measure if issu...
[08:38:19] <wikibugs>	 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Joe) I also noticed that the number of requests for `/v3/precache` has increased a lot around the time of the issue. This points to Change-Pr...
[18:10:46] <wikibugs>	 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Ladsgroup) >>! In T263910#6507925, @Joe wrote: >  > It seems like ORES right now is operating almost "at-capacity",  and the additional traff...
[21:32:47] <wikibugs>	 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10RBrounley_WMF) We’re working to patch up our end, switching to streams with querying the Ores api when streams fail. Sorry will update soon