[00:06:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space thanos-be2006:9100:/ 1.078% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[00:16:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space thanos-be2006:9100:/ 0% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[03:32:35] <jinxer-wm>	 FIRING: DiskSpace: Disk space thanos-be1005:9100:/ 3.805% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be1005 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[03:38:50] <jinxer-wm>	 FIRING: DiskSpace: Disk space ms-be1066:9100:/srv/swift-storage/sdb3 1.794% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=ms-be1066 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[05:54:00] <jinxer-wm>	 FIRING: SystemdUnitFailed: prometheus-dpkg-success-textfile.service on thanos-be1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:55:54] <Emperor>	 cleared up the disk-fill on thanos-be1005 (this is T423690 again)
[06:55:55] <stashbot>	 T423690: Thanos backends filling their root filesystems overnight - https://phabricator.wikimedia.org/T423690
[06:57:35] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space thanos-be1005:9100:/ 0.1804% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be1005 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[06:59:00] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: prometheus-dpkg-success-textfile.service on thanos-be1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:25:39] <Emperor>	 ran a bulk-vacuum on ms-be1066, now only 59% used
[21:11:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space thanos-be2006:9100:/ 3.854% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[23:04:00] <jinxer-wm>	 FIRING: SystemdUnitFailed: prometheus-dpkg-success-textfile.service on thanos-be2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed