[06:45:12] <jinxer-wm>	 FIRING: ThanosCompactHalted: Thanos Compact on titan2001 has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted
[10:00:12] <jinxer-wm>	 RESOLVED: ThanosCompactHalted: Thanos Compact on titan2001 has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted
[10:02:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: thanos-compact.service on titan2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:04:23] <tappof>	 ^^ it's me
[10:42:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: thanos-compact.service on titan2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:33:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[12:59:35] <jinxer-wm>	 FIRING: DiskSpace: Disk space kafka-logging1001:9100:/srv 3.893% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[13:13:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[13:19:35] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space kafka-logging1001:9100:/srv 3.902% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[13:21:35] <jinxer-wm>	 FIRING: DiskSpace: Disk space kafka-logging1001:9100:/srv 3.894% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[13:51:35] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space kafka-logging1001:9100:/srv 3.698% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace