[02:15:41] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db2230:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:11:46] federico3: o/ [06:15:41] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db2230:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:34:33] federico3: could you ack the alerts on icinga for db-test2001 and 2002, as well as db2230? thanks! [08:35:23] uhm is it showing on alerts.w.o? I'm not seeing it re [08:35:27] *red [08:38:59] I don't know [08:41:31] aha, found it [08:41:38] thanks! [08:42:20] (we should really find a way to "tag" the mariadb-data-persistence alerts) [08:43:19] it is called team=data_persistence :-) [08:46:07] ah, TIL team=data-persistence [08:46:57] do we know if https://alerts.wikimedia.org/?q=%40cluster%3Dwikimedia.org&q=team%3Ddata-persistence is catching everything? [08:58:25] that's for you to configure it properly on puppet :-D [08:59:10] don't ask what data persistence can do for you, ask what you can do for data persistence :-DDD [16:29:34] BTW, the name is no longer Minio, now it is called AIStor