[00:10:34] FIRING: DiskSpace: Disk space arclamp1001:9100:/srv 3.896% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:15:34] FIRING: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 3.804% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:20:34] RESOLVED: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 3.765% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:22:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [02:35:08] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [02:35:35] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 2.81% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:57:40] RESOLVED: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [03:25:53] FIRING: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:30:53] RESOLVED: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:00:35] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.857% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:10:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.966% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:15:34] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.875% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:20:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.919% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:40:34] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.894% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:35:08] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [06:40:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.899% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:05:34] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.896% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:11:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.934% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:11:34] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.879% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:21:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.902% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:22:00] ^^ looking [09:51:10] ^^ mostly due to the dc switchover https://w.wiki/foCf [10:35:08] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [13:21:34] FIRING: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.487% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:35:08] FIRING: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [15:21:40] FIRING: LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [15:26:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [15:31:40] RESOLVED: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [15:41:35] RESOLVED: DiskSpace: Disk space kafka-logging1006:9100:/srv 3.145% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=kafka-logging1006 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:51:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [17:10:08] RESOLVED: [2x] SlothSliErrorRatioRateAbsent: SLI error ratio rate is absent (linkrecommendation / requests-eqiad) - TODO - https://alerts.wikimedia.org/?q=alertname%3DSlothSliErrorRatioRateAbsent [17:16:38] ^^ https://gitlab.wikimedia.org/repos/sre/slothslos/-/merge_requests/63 [19:51:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [20:43:34] FIRING: DiskSpace: Disk space arclamp1001:9100:/srv 3.901% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=arclamp1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:48:34] FIRING: [2x] DiskSpace: Disk space arclamp1001:9100:/srv 3.836% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:11:40] RESOLVED: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag