[02:18:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.987% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:03:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.882% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:52:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.986% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:02:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.579% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:58:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.986% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:03:34] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.768% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:09:34] FIRING: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.986% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:34:34] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.711% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:39:35] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on alert2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:39:37] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on alert2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:39:55] FIRING: SystemdUnitFailed: check_icinga-alert2002.service on alert2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:42:28] FIRING: [2x] KeyholderUnarmed: 1 unarmed Keyholder key(s) on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:43:35] o/ I am merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1369312 and 1369313 to test out the icinga-less vm setup we talked about last week [08:44:55] RESOLVED: [2x] SystemdUnitFailed: check_icinga-alert1002.service on alert2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:45:10] FIRING: [3x] SystemdUnitFailed: check_icinga-alert1002.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:47:28] RESOLVED: [2x] KeyholderUnarmed: 1 unarmed Keyholder key(s) on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:49:55] RESOLVED: [3x] SystemdUnitFailed: check_icinga-alert1002.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:04:27] FIRING: [2x] DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.876% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:49:27] RESOLVED: DiskSpace: Disk space prometheus2007:9100:/srv/prometheus/k8s-dse 3.767% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=prometheus2007 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:49:51] Happy Monday, I want to merge a couple of patches to remove metrics with cardinality explosions. Is there a place that I can check for health of the infra? I like to see nice graphs :D [09:51:43] I've never used it, but https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring seems promising :-P [09:55:24] Yeah, Amir1, I was just about to share the same link when I noticed that something was broken in some of the graphs.. I'm checking .. [09:55:59] ah thanks! [09:56:46] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1369624 Once I get +1 from DBAs, I'll push this [09:57:16] This is also a nice drop and fully useless it seems https://gerrit.wikimedia.org/r/c/operations/puppet/+/1369623 [10:30:50] ok Amir1, I've recovered the cardinality exporters, and the graphs are now being drawn.. [10:31:24] oh nice [10:31:32] > https://gerrit.wikimedia.org/r/c/operations/puppet/+/1369623 [10:31:36] For this one, I gave it a -1 because the metric is currently used by a dashboard.. [10:33:05] If I understood it correctly, the metric simply should change to count1? [10:35:31] tappof: they are already actually using count1 [10:35:39] which is a different metric :P [10:35:45] aaaah sorry Amir1 [10:36:18] who names a variable count1 :D [11:13:16] tappof: oh also there is this one: https://phabricator.wikimedia.org/T444937 I was thinking of doing this https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1369596 but not 100% sure, I went back and forth between different solutions [15:36:03] tappof: I see some decent dent in https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?from=now-3h&to=now&timezone=utc&var-prometheus=ops&var-site=eqiad but can't be sure if it's related or not [15:36:53] FIRING: ProbeDown: Service webperf2003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:41:53] RESOLVED: ProbeDown: Service webperf2003:443 has failed probes (http_performance_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#webperf2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:44:05] Amir1: There has been a drop (https://w.wiki/i$B2), but I believe the one you can see in the graph you shared is still unrelated. [15:44:12] This is because those metrics relate to the currently open TSDB block, and they're reset only when the block is closed and a new one is opened. [15:45:02] It happens every two hours, so we'll see that most of the MySQL-related series will be missing from the new one. [15:45:22] ah thanks [16:55:10] FIRING: [2x] SystemdUnitFailed: check_icinga-alert1002.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:55:55] FIRING: [3x] SystemdUnitFailed: check_icinga-alert1002.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:00:10] RESOLVED: [3x] SystemdUnitFailed: check_icinga-alert1002.service on alert1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:47:54] hello friends! We're starting the process of adding SBOMs to our container pipelines, and I think it'd be useful to have a metric of SBOM availability we can observe over time [17:48:47] the rough process for this looks like 1) get all the pods running in a cluster that use our registry, 2) talk to the registry and see if there's an SBOM in the manifest, and 3) report the number [17:49:29] my instinct is a cronjob on k8s and pushgateway, but I'd love more opinions [18:14:40] FIRING: LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?var-cluster=logging-eqiad&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [18:19:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [19:13:00] lizimedia: interesting! yes offhand the push gateway sounds like it'd be a good fit. but I would say lets spin up a task if possible and we can do a bit more reasoning about the different options there [19:17:05] herron: https://phabricator.wikimedia.org/T445130 this is a little light on details rn, but it's a start [19:18:32] running on k8s seems like the easiest options, since there's a lot of mechanisms for k8s clusters to expose their own APIs to running pods [19:42:16] lizimedia: sounds good, just replied on task, yeah I totally think push gateway is the way to go for the onboarding goal tracking metrics [22:19:40] FIRING: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag [22:39:40] RESOLVED: [2x] LogstashKafkaConsumerLag: Too many messages in logging-eqiad for group logstash7-codfw - https://wikitech.wikimedia.org/wiki/Logstash#Kafka_consumer_lag - https://alerts.wikimedia.org/?q=alertname%3DLogstashKafkaConsumerLag