[02:19:56] FIRING: SystemdUnitFailed: prometheus-mysqld-exporter.service on db2250:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:43:34] FIRING: DiskSpace: Disk space thanos-be2008:9100:/srv/swift-storage/accounts0 0.1945% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:19:56] FIRING: SystemdUnitFailed: prometheus-mysqld-exporter.service on db2250:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:29:33] ^ fixed, it was because of yesterday's reimage [06:34:42] RESOLVED: SystemdUnitFailed: prometheus-mysqld-exporter.service on db2250:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:43:34] FIRING: DiskSpace: Disk space thanos-be2008:9100:/srv/swift-storage/accounts0 0.1938% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:51:15] that's odd [08:51:44] Hm, three copies of a quarantined container. [09:04:42] FIRING: SystemdUnitFailed: swift_dispersion_stats_lowlatency.service on ms-fe1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:14:42] RESOLVED: SystemdUnitFailed: swift_dispersion_stats_lowlatency.service on ms-fe1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:28:35] RESOLVED: DiskSpace: Disk space thanos-be2008:9100:/srv/swift-storage/accounts0 0.1934% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace