[02:15:11] <jinxer-wm>	 FIRING: [2x] PrometheusZombieSeriesDetected: Zombie series detected on k8s (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusZombieSeriesDetected
[06:15:11] <jinxer-wm>	 FIRING: [2x] PrometheusZombieSeriesDetected: Zombie series detected on k8s (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusZombieSeriesDetected
[10:15:12] <jinxer-wm>	 FIRING: [2x] PrometheusZombieSeriesDetected: Zombie series detected on k8s (codfw) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusZombieSeriesDetected
[14:50:14] <tappof>	 ^^ I quickly checked the dashboard https://grafana.wikimedia.org/goto/afo889afsiayof?orgId=1, and it seems that starting from the first days of May, the deployments have affected the zombie series detection metric a bit more than usual.
[14:50:24] <tappof>	 I'll take a deeper look next week; however, I've already checked the size of the Thanos index headers, and they do not seem to be impacted.
[14:50:53] <tappof>	 Alerts have been acknowledged.
[15:47:26] <jinxer-wm>	 RESOLVED: PrometheusZombieSeriesDetected: Zombie series detected on k8s (eqiad) - https://wikitech.wikimedia.org/wiki/Prometheus#Runbooks - https://grafana.wikimedia.org/d/taff979/prometheus-tsdb-cardinality-monitoring?orgId=1&from=now-14d&to=now&timezone=utc&var-prometheus=k8s&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DPrometheusZombieSeriesDetected