[00:09:56] RESOLVED: [2x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:10:53] FIRING: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:15:53] RESOLVED: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:17:11] FIRING: [2x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:36:53] FIRING: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:37:11] RESOLVED: [2x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:02:11] FIRING: [3x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:04:56] RESOLVED: [3x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:09:56] FIRING: [4x] SystemdUnitFailed: puppet-agent-timer.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:15:02] RESOLVED: [4x] SystemdUnitFailed: puppet-agent-timer.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:50:02] FIRING: [2x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:52:11] RESOLVED: [2x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:32:11] FIRING: [3x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:34:56] RESOLVED: [3x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:51:53] RESOLVED: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:53:53] FIRING: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:03:53] RESOLVED: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:04:56] FIRING: [3x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:05:53] FIRING: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:07:11] RESOLVED: [3x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:20:53] RESOLVED: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:22:53] FIRING: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:37:53] RESOLVED: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:39:53] FIRING: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:44:53] RESOLVED: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:44:56] FIRING: [3x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:47:11] FIRING: [3x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:44:56] RESOLVED: [2x] SystemdUnitFailed: user@17187.service on webperf1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:59:34] FIRING: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:04:35] RESOLVED: DiskSpace: Disk space prometheus1008:9100:/srv/prometheus/k8s-dse 3.989% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:43:48] FIRING: PuppetDisabled: Puppet disabled on kafkamon2003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [12:58:48] FIRING: [2x] PuppetDisabled: Puppet disabled on kafkamon1003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [16:58:48] FIRING: [2x] PuppetDisabled: Puppet disabled on kafkamon1003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [18:18:40] FIRING: LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=40&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures [18:23:40] RESOLVED: LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=40&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures [20:35:53] FIRING: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:40:53] RESOLVED: ProbeDown: Service webperf1003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:58:48] FIRING: [2x] PuppetDisabled: Puppet disabled on kafkamon1003:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled