[06:46:25] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1022:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:46:25] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1022:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:56:58] I'm going to mute that alert for 44 hours (until 08:00 on Monday) because getting spammed all weekend is annoying [19:19:18] > s6 eqiad snapshot wrong_size 17 hours ago 478.5 GB -5.9 % The previous backup had a size of 508.8 GB, a change larger than 5.0%. [19:19:26] Text table clean up [20:04:49] Thanks Emperor [20:04:59] Amir1: good job!!! [22:32:48] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db2212:9104 has too large replication lag (11m 6s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2212&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat