[08:05:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:15:40] <jinxer-wm>	 FIRING: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[11:20:40] <jinxer-wm>	 RESOLVED: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[12:05:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:28:47] <inflatador>	 hey 0lly, I'm getting a silent failure from CI on https://gerrit.wikimedia.org/r/c/operations/alerts/+/1130114 . when I run promtool locally, I get `yaml: unmarshal errors: line 3: field groups not found in type main.unitTestFile` . Any suggestions on this one? I don't see `groups:` in any unit test files, but maybe I missed something
[13:31:46] <godog>	 inflatador: interesting, I suspect it might be a promtool / prometheus version discrepancy, what version are you running locally? does it work locally if you run tests in the container instead (instructions in README)
[13:31:47] <inflatador>	 nm, I was pointing to the wrong file ;(
[13:32:03] <godog>	 lol ok, I also was about to ask what you mean by silent failure
[13:32:25] <inflatador>	 godog in other words, doing `promtool test rules team-data-platform/rdf_streaming_updater_global.yaml` instead of `promtool test rules team-data-platform/rdf_streaming_updater_global_test.yaml` . No sure about the CI though
[13:32:48] <inflatador>	 But if I point to the right file, I do get an error that I can work with
[13:33:11] <godog>	 ack, yeah I just run 'tox'
[13:33:17] <godog>	 which is what ci does anyways
[13:34:16] <inflatador>	 ACK, thanks for the advice...alert is passing now ;)
[14:32:40] <jinxer-wm>	 FIRING: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[14:37:40] <jinxer-wm>	 RESOLVED: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[15:35:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: curator_actions_cluster_wide.service on logging-sd1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed