[00:00:48] 10Scoring-platform-team: [Investigate] Does alarm timeout break Celery? - https://phabricator.wikimedia.org/T175875#3606473 (10awight) This might be invalid, I need to set more appropriate limits and see how the pool behaves. [04:03:39] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, 10User-Ladsgroup: Review and fix file handle management in worker and celery processes - https://phabricator.wikimedia.org/T174402#3606687 (10awight) I think I've been wrong all this time. The system's total open file count is... [04:18:17] 10Scoring-platform-team: Celery task pool doesn't degrade nicely. - https://phabricator.wikimedia.org/T175875#3606708 (10awight) [05:59:19] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review, 10User-Ladsgroup: [Discuss] Make ORES Review Tool preferences more prominent - https://phabricator.wikimedia.org/T167910#3606840 (10Johan) [07:53:19] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, 10User-Ladsgroup: Review and fix file handle management in worker and celery processes - https://phabricator.wikimedia.org/T174402#3606989 (10akosiaris) > ores1002 limits are, > ``` > ulimit -a > ... > open files... [15:41:18] 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017): Very long search times on RC Page for "Very likely good faith" + "Likely have problems" (on en.wiki only?) - https://phabricator.wikimedia.org/T164796#3608233 (... [16:06:07] awight: hows it going? [18:25:20] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:38:00] Amir1: i dont know if you are about but this error looks legit ^^ [18:53:50] RECOVERY - puppet on ores-lb-02 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures