[10:03:23] <Lycurgus>	 so it's all ml type stuff?
[13:26:23] <wikibugs>	 10ORES, 10Scoring-platform-team (Current): Address icinga noise from wmflabs - https://phabricator.wikimedia.org/T231222 (10JHedden) I've looked at packet captures and traced processes on both ends, as far as I can tell the metrics are being sent and stored correctly in graphite's whisper database.   I spot ch...
[15:47:47] <wikibugs>	 10ORES, 10Scoring-platform-team (Current): Address icinga noise from wmflabs - https://phabricator.wikimedia.org/T231222 (10Halfak) OK!  So I [cut down the number of web workers per node from 48 to 36](https://wikitech.wikimedia.org/w/index.php?title=Hiera:Ores&diff=1836902&oldid=1826595) and I see both the oo...
[15:48:41] <halfak>	 icinga-wm is back.  Let's see if it is still noisy. 
[15:49:22] <icinga-wm>	 PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/ORES
[15:49:40] <halfak>	 Hmm
[15:50:19] <halfak>	 looks like the issue is back. 
[15:50:30] <halfak>	 I told icinga to shuddup again. 
[15:59:02] <halfak>	 This is driving me crazy. 
[15:59:09] <halfak>	 I just can't figure it out!
[15:59:35] <halfak>	 I'm checking staging again. 
[15:59:43] <halfak>	 I want to make sure I can't replicate the problem there. 
[17:26:14] <xinbenlv>	 @halfak, follow up with the question.have you considered generate and cache all ORES for every revision?
[17:28:20] <xinbenlv>	 (or is the resource capacity allowing this?)
[17:29:29] <halfak>	 Hey xinbenlv.  Yes!  We've got a task for storing "revision-score" events.  
[17:29:33] <halfak>	 We don't have dumps yet. 
[17:33:11] <xinbenlv>	 Do you have an issue?
[17:33:24] <xinbenlv>	 I mean, is there a phabricator issue I can subscribe to?
[17:47:50] <halfak>	 https://phabricator.wikimedia.org/T209739
[17:47:53] <halfak>	 xinbenlv, ^