[03:08:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.863% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:58:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.79% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:03:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.845% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:03:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.767% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:03:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.799% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:08:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.716% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:08:50] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.583% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:34:48] FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:44:48] RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:08:50] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.494% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:08:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.77% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:48:35] FIRING: [2x] DiskSpace: Disk space prometheus1007:9100:/srv/prometheus/k8s-dse 3.733% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:34:48] FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:44:48] RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure