[00:11:34] 10ORES, 10Machine Learning Platform, 10Operations, 10serviceops, 10Wikimedia-production-error: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Ladsgroup) Looking at the pattern of requests to ores in the past couple of days, it seems OKAPI has been bringing dow... [00:12:19] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Reedy) [06:19:15] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Joe) >>! In T263910#6503699, @Ladsgroup wrote: > Looking at the pattern of requests to ores in the past couple of days, it seems OKAPI has be... [07:39:57] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10akosiaris) >>! In T263910#6503978, @Joe wrote: >>>! In T263910#6503699, @Ladsgroup wrote: >> Looking at the pattern of requests to ores in th... [09:40:13] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Ladsgroup) This sounds good to me, my only worry is that it would be too much work for something that's going to be replaced in the soon-ish... [11:06:32] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10awight) #okapi team, please consider following the https://stream.wikimedia.org/?doc#/Streams/get_v2_stream_revision_score event stream rathe... [16:40:08] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10RBrounley_WMF) Hey, connecting with folks on ORES team around this today - sorry we were given advice that the ORES stream may have some data... [16:46:51] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Joe) >>! In T263910#6505891, @RBrounley_WMF wrote: > Hey, connecting with folks on ORES team around this today - sorry we were given advice t... [16:55:19] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10Ladsgroup) >>! In T263910#6505891, @RBrounley_WMF wrote: > Hey, connecting with folks on ORES team around this today - sorry we were given ad... [17:28:44] 10ORES, 10Machine Learning Platform (Current): Script to restart ORES uwsgi service - https://phabricator.wikimedia.org/T264121 (10calbon) I'm going to close this ticket because the issue is being caused by #okapi and will be resolved soon enough that this script isn't required. But for future Chris: Fabric... [17:28:57] 10ORES, 10Machine Learning Platform (Current): Script to restart ORES uwsgi service - https://phabricator.wikimedia.org/T264121 (10calbon) 05Open→03Resolved [19:11:17] 10ORES, 10Machine Learning Platform, 10Okapi, 10Operations, and 2 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10ACraze) >>! In T263910#6505937, @Ladsgroup wrote: > If there are scores missing from the stream, feel free to hit the endpoint but I suggest... [23:48:26] Hi is ORES is experiencing some issues with regards to Redis exceptions. Were intermittently seeing issues that started around on 9/24. The last error we got was a 500 trying to get these revisions: [23:48:28] https://ores.wikimedia.org/v3/scores/frwiki/?models=damaging&revids=175182681|175182689 [23:50:57] sorry i was DCed but i attemped to ask: is ORES experiencing some issues with regards to Redis exceptions. Were intermittently seeing issues that started around on 9/24. The last error we got was a 500 trying to get these revisions: [23:51:11] https://ores.wikimedia.org/v3/scores/frwiki/?models=damaging&revids=175182681|175182689