[00:22:34] RESOLVED: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 3.749% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:33:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [02:35:07] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [04:33:41] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [05:10:56] FIRING: PrometheusSeriesCountAnomalyHigh: Anomalous increase in number of active series on k8s (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=k8s&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCountAnomalyHigh [06:35:07] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [08:33:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [09:10:56] FIRING: PrometheusSeriesCountAnomalyHigh: Anomalous increase in number of active series on k8s (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=k8s&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCountAnomalyHigh [10:35:08] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [12:15:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.926% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:25:34] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.861% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:33:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [13:03:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.876% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:08:35] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.893% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:10:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.961% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:10:56] FIRING: PrometheusSeriesCountAnomalyHigh: Anomalous increase in number of active series on k8s (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=k8s&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCountAnomalyHigh [14:15:34] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.853% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:16:04] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.938% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:20:50] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.884% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:23:40] RESOLVED: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [14:25:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.915% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:30:34] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.835% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:35:08] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [14:35:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.957% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:35:56] RESOLVED: PrometheusSeriesCountAnomalyHigh: Anomalous increase in number of active series on k8s (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=k8s&var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DPrometheusSeriesCountAnomalyHigh [18:22:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [18:35:08] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [18:35:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.577% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:22:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [22:35:08] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [22:35:35] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.263% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:02:40] FIRING: LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=40&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures [23:07:40] RESOLVED: LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=40&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures