[00:46:11] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: curator_actions_cluster_wide.service on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:48:41] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:48:41] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:10:55] <jinxer-wm>	 FIRING: LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=40&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[12:15:55] <jinxer-wm>	 RESOLVED: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors  - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures
[12:48:41] <jinxer-wm>	 FIRING: SystemdUnitFailed: curator_actions_cluster_wide.service on logstash1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:13:40] <arnaudb>	 hello o11y! I come to ask a classic IT question: why does it work on my computer? https://integration.wikimedia.org/ci/job/alerts-pipeline-test/1987/execution/node/64/log/ :D I don't understand how I'm able to pint/promtool test locally but CI rejects the same lines
[14:11:43] <godog>	 arnaudb: interesting, what is the related change? and running 'tox' locally works in that case ?
[14:11:54] <godog>	 I'll silence the curator alerts
[14:38:58] <arnaudb>	 sorry godog I was in and out of meetings, the change is https://gerrit.wikimedia.org/r/c/operations/alerts/+/1050498 and tox locally fails on varnishkafka, but not on my changes
[14:39:49] <godog>	 arnaudb: sure no problem, fascinating though do you have the local tox output ?
[14:40:05] * arnaudb runs a new tox -p
[14:41:36] <arnaudb>	 here you go https://www.irccloud.com/pastebin/YofKkle6/tox%20-p
[14:48:50] <godog>	 thank you, I can't reproduce the error locally, what version of pint is that arnaudb ? 'pint version' to find out
[14:49:25] <arnaudb>	 pint: 0.60.0 (revision: 7f42c894778636d87ec69a2d0d889b5b18a2f1fa), promtool: version 2.53.0 (branch: HEAD, revision: 4c35b9250afefede41c5f5acd76191f90f625898)
[14:53:22] <godog>	 arnaudb: yep it is the pint version indeed, I can reproduce with 0.65 but not 0.40 which is the default we're using, my recommendation for now would be to build and use the docker container instead as per README.md
[14:53:37] <arnaudb>	 ack, thanks godog !
[14:53:42] <godog>	 sure np