[00:00:40] RESOLVED: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [03:53:49] FIRING: [2x] PuppetDisabled: Puppet disabled on kafkamon1003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [07:53:49] FIRING: [2x] PuppetDisabled: Puppet disabled on kafkamon1003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [08:40:07] FIRING: SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://grafana.wikimedia.org/d/slot-pilot-slo-detail/sloth-slo-detail?var-sli_window=5m&var-service=linkrecommendation&var-slo=requests-eqiad - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [10:35:07] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [10:48:49] FIRING: [2x] PuppetDisabled: Puppet disabled on kafkamon1003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [10:58:49] RESOLVED: [2x] PuppetDisabled: Puppet disabled on kafkamon1003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [14:35:07] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [16:00:40] FIRING: LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [16:05:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [17:04:48] FIRING: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:14:48] RESOLVED: PuppetFailure: Puppet has failed on logging-hd2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:35:07] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [20:05:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [20:35:40] RESOLVED: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [22:06:34] FIRING: DiskSpace: Disk space arclamp1001:9100:/srv 3.92% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:16:34] FIRING: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 3.766% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:21:34] RESOLVED: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 3.751% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:35:07] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [22:38:34] FIRING: DiskSpace: Disk space arclamp1001:9100:/srv 3.922% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:48:34] FIRING: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 3.776% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace