[00:45:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: grafana-ldap-users-sync.service on grafana1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:45:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: grafana-ldap-users-sync.service on grafana1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:00:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: grafana-ldap-users-sync.service on grafana1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:19:25] <XioNoX>	 hello! is it possible for all the alerts with team=netops to instead (or in addition) have `scope=network` ? https://alerts.wikimedia.org/?q=team%3Dnetops
[13:17:57] <tappof>	 Hey XioNoX, are we talking about alerts from Icinga?
[13:24:50] <XioNoX>	 tappof: all the ones that show up on that link
[14:55:55] <swfrench-wmf>	 [non-urgent] following up on my question about prometheus-jmx-exporter yesterday: it turns out this does not block debian upgrades - i.e., deb11u1 packages have been imported into bookworm- and trixie-wikimedia, since they're just jars (and thus trivially work). many thanks to m.oritzm for pointing this out :)
[15:44:41] <tappof>	 ack XioNoX, I'll take a look
[17:04:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[17:14:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[19:47:43] <jinxer-wm>	 FIRING: [2x] BenthosKafkaConsumerLag: Too many messages in jumbo-eqiad for group benthos-webrequest_live - TODO  - https://alerts.wikimedia.org/?q=alertname%3DBenthosKafkaConsumerLag
[20:52:43] <jinxer-wm>	 RESOLVED: [2x] BenthosKafkaConsumerLag: Too many messages in jumbo-eqiad for group benthos-webrequest_live - TODO  - https://alerts.wikimedia.org/?q=alertname%3DBenthosKafkaConsumerLag