[00:04:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space centrallog2002:9100:/srv 3.947% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[03:30:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: generate-mysqld-exporter-config.service on prometheus1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:30:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: generate-mysqld-exporter-config.service on prometheus1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:05:13] <tappof>	 ^^ T410365
[09:05:13] <stashbot>	 T410365: SystemdUnitFailed: generate-mysqld-exporter-config.service on prometheus100[56]:9100 - https://phabricator.wikimedia.org/T410365
[09:15:25] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: generate-mysqld-exporter-config.service on prometheus1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[18:20:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space centrallog2002:9100:/srv 3.979% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[20:20:03] <herron>	 I've just grown /srv on centrallog2002 to 3T, this alert should clear in a moment
[20:20:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space centrallog2002:9100:/srv 3.496% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=centrallog2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[20:21:00] <denisse>	 Thanks!!