[04:42:22] <wikibugs>	 10Scoring-platform-team, 10Gadgets, 10Code-Health, 10artificial-intelligence: Detect/flag potentially malicious gadget/javascript edits - https://phabricator.wikimedia.org/T208140 (10Bawolff) > Special:RecentChanges filtered to JS pages  If the JS is truly malicious, its probably too late at that point unl...
[12:29:59] <Amir1>	 o/
[13:31:22] <wikibugs>	 (03PS1) 10Ladsgroup: Untrack non-wheel files from git lfs [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/472650 (https://phabricator.wikimedia.org/T209060)
[13:32:30] <wikibugs>	 (03CR) 10Ladsgroup: [C: 032] Untrack non-wheel files from git lfs [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/472650 (https://phabricator.wikimedia.org/T209060) (owner: 10Ladsgroup)
[13:35:06] <wikibugs>	 (03PS1) 10Ladsgroup: Bump wheels to HEAD [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/472651
[13:38:08] <wikibugs>	 (03CR) 10Ladsgroup: [V: 032 C: 032] Bump wheels to HEAD [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/472651 (owner: 10Ladsgroup)
[13:53:03] <Amir1>	 akosiaris: Hey, around for a quick deployment for the git lfs issue?
[13:58:52] <wikibugs>	 10ORES, 10Scoring-platform-team (Current), 10Patch-For-Review, 10User-Ladsgroup: ores returns error for draftquality models - https://phabricator.wikimedia.org/T209060 (10Ladsgroup) Deployed the change on beta, it works. Will deploy it later next week.
[14:35:33] <akosiaris>	 Amir1: yeah sure
[14:35:38] <akosiaris>	 I am now
[14:36:07] <Amir1>	 let's do it
[14:36:19] <akosiaris>	 same procedure as last time ?
[14:49:06] <wikibugs>	 (03CR) 10Ladsgroup: [V: 032 C: 032] Increase number of parallel connections to 9 [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/472413 (https://phabricator.wikimedia.org/T191842) (owner: 10Ladsgroup)
[15:09:50] <wikibugs>	 10Scoring-platform-team (Current), 10Operations, 10Release-Engineering-Team, 10Scap, and 2 others: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842 (10Ladsgroup) With the new number of 9 parallel connection and 14 minutes to deploy (down from around half...
[16:34:18] <wikibugs>	 10ORES, 10Scoring-platform-team, 10monitoring, 10Wikimedia-Incident: Create Grafana graph to show number of ORES API requests per user-agent - https://phabricator.wikimedia.org/T182222 (10Ladsgroup) Given that we have the 4 parallel connection per IP in place and difficulties of implementing this. I propos...
[17:02:02] <wikibugs>	 10ORES, 10Scoring-platform-team, 10Operations, 10Performance: Diagnose and fix 4.5k req/min ceiling for ores* requests - https://phabricator.wikimedia.org/T182249 (10Halfak) If we could handle twice the capacity, then we could allow researchers to query us twice as fast.  :)  We could lift our simultaneous...
[17:24:14] <wikibugs>	 10ORES, 10Scoring-platform-team, 10Operations, 10Performance: Diagnose and fix 4.5k req/min ceiling for ores* requests - https://phabricator.wikimedia.org/T182249 (10Ladsgroup) We can't increase number of our celery workers due to memory allocation. Maybe we need to tackle {T182350} then fix things and the...
[18:55:41] <wikibugs>	 10ORES, 10Scoring-platform-team, 10Operations, 10Performance: Diagnose and fix 4.5k req/min ceiling for ores* requests - https://phabricator.wikimedia.org/T182249 (10Halfak) It looks like we do have a bit more ceiling for memory usage.  I half-remembered us tuning our worker-count down due to issues with c...