[00:42:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space centrallog2002:9100:/srv 3.977% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[08:14:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: statograph_post.service on alert2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:19:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: statograph_post.service on alert2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:27:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space centrallog2002:9100:/srv 3.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[15:47:03] <elukey>	 hey folks!
[15:47:43] <elukey>	 I'd need to set up an alarm related to a minimal set of rows that a file must have on puppetservers
[15:48:07] <elukey>	 (it is basically the result of an airflow job that pushes the data to puppetservers, and then the data is distributed to the whole CDN)
[15:48:37] <elukey>	 the only way that I can think of is to create a systemd timer that pushes a metric via the host's exporter, or similar
[15:48:39] <elukey>	 and then alarm on it
[15:48:44] <elukey>	 are there better ways to do it?
[16:49:27] <cwhite>	 Easiest is probably the timer writing a prom file for the host exporter to serve.  A slightly nicer alternative is a timer that pushes the data to PushGateway.
[17:18:57] <hnowlan>	 elukey: there's prometheus::node_textfile, you could do some shell golf in run_cmd. already has support for the timer 
[19:27:49] <jinxer-wm>	 FIRING: DiskSpace: Disk space centrallog2002:9100:/srv 2.796% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[20:12:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space centrallog2002:9100:/srv 3.373% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[23:13:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space centrallog2002:9100:/srv 3.971% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace