[09:33:33] <Amir1>	 o/
[12:42:23] <icinga-wm>	 PROBLEM - ores-extension grafana alert on icinga1001 is CRITICAL: CRITICAL: ORES extension ( https://grafana.wikimedia.org/d/000000263/ores-extension ) is alerting: Service hits for obtaining thresholds alert.
[12:43:37] <icinga-wm>	 RECOVERY - ores-extension grafana alert on icinga1001 is OK: OK: ORES extension ( https://grafana.wikimedia.org/d/000000263/ores-extension ) is not alerting.
[12:46:59] <icinga2-wm>	 PROBLEM - puppet on ORES-worker02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:14:59] <icinga2-wm>	 RECOVERY - puppet on ORES-worker02.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:11:33] <Amir1>	 okay, I go eat something
[15:03:25] <wikibugs>	 10ORES, 10Scoring-platform-team, 10Upstream: Make ORES dependency solving upstreamable - https://phabricator.wikimedia.org/T201657 (10Ladsgroup) p:05Triage→03Low
[15:03:32] <wikibugs>	 10ORES, 10Scoring-platform-team: User-facing tool to explain scoring algorithms - https://phabricator.wikimedia.org/T204560 (10Ladsgroup) p:05Triage→03Low
[15:07:57] <wikibugs>	 10Scoring-platform-team, 10Wikilabels: Internationalize static HTML - https://phabricator.wikimedia.org/T185626 (10Ladsgroup) p:05Triage→03Low
[15:08:14] <wikibugs>	 10ORES, 10Scoring-platform-team, 10Graphite: Add a graph of ORES Celery task queue length - https://phabricator.wikimedia.org/T182914 (10Ladsgroup) The best way to handle it is to have a Prometheus exporter for celery. I found this already: https://github.com/zerok/celery-prometheus-exporter @akosiaris Do yo...
[15:08:16] <wikibugs>	 10ORES, 10Scoring-platform-team, 10Graphite: Add a graph of ORES Celery task queue length - https://phabricator.wikimedia.org/T182914 (10Ladsgroup) p:05Triage→03Normal
[15:08:20] <wikibugs>	 10Scoring-platform-team, 10edittypes-modeling, 10User-Ladsgroup, 10artificial-intelligence: Deploy pilot edittype model - https://phabricator.wikimedia.org/T196634 (10Ladsgroup) p:05Triage→03Lowest
[15:08:22] <wikibugs>	 10Scoring-platform-team, 10ORES-Support-Checklist: Support for languages in ores-support-checklist - https://phabricator.wikimedia.org/T191768 (10Ladsgroup) p:05Triage→03Lowest
[15:08:24] <wikibugs>	 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for basque - https://phabricator.wikimedia.org/T184205 (10Ladsgroup) p:05Triage→03Low
[15:08:41] <wikibugs>	 10Scoring-platform-team, 10ORES-Support-Checklist: Make a bot that updates mediawiki page using ores-support-checklist automatically - https://phabricator.wikimedia.org/T191765 (10Ladsgroup) p:05Triage→03Lowest
[17:02:23] <hoo>	 The sync is not happening, is itß
[17:02:25] <hoo>	 *?
[17:34:36] <Amir1>	 hoo: no, today is WMF holiday
[17:39:41] <wikibugs>	 10Scoring-platform-team (Current), 10editquality-modeling, 10Spanish-Sites, 10User-Ladsgroup, 10artificial-intelligence: Train/test damaging & goodfaith models for Spanish Wikiquote (eswikiquote) - https://phabricator.wikimedia.org/T209784 (10Ladsgroup) p:05Triage→03High a:03Ladsgroup
[18:21:40] <apergos>	 ah so it is
[20:05:35] <hoo>	 Amir1: "revscoring.errors.ModelConsistencyError: ModelConsistencyError: Expected labels {'attack'} represented" Does that ring any bells for you offhand?
[20:07:14] <Amir1>	 hoo: The only thing I can think is that 'attack' is a label on draftquality models
[20:07:23] <Amir1>	 it should not appear in other models
[20:07:34] <hoo>	 Amir1: Yeah, I'm working with draftquality
[20:09:08] <Amir1>	 hoo: if so, it might mean one of the data doesn't have the label properly set. 
[20:16:03] <hoo>	 Amir1: I guess I was missing the --labels param to revscoring cv_tune
[20:17:47] <Amir1>	 hmm, it's very likely
[22:17:46] <wikibugs>	 (03PS3) 10Krinkle: ext.ores.highlighter: Fix minor memory alloc [extensions/ORES] - 10https://gerrit.wikimedia.org/r/485139
[22:17:49] <wikibugs>	 (03PS4) 10Krinkle: ext.ores.highlighter: Fix minor memory alloc [extensions/ORES] - 10https://gerrit.wikimedia.org/r/485139
[22:17:53] <wikibugs>	 (03CR) 10Krinkle: "(untested)" (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/485139 (owner: 10Krinkle)
[22:30:20] <wikibugs>	 10Scoring-platform-team (Current), 10editquality-modeling, 10Spanish-Sites, 10User-Ladsgroup, 10artificial-intelligence: Train/test damaging & goodfaith models for Spanish Wikiquote (eswikiquote) - https://phabricator.wikimedia.org/T209784 (10Ladsgroup) https://github.com/wikimedia/editquality/pull/178