[05:29:01] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on pc2012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:42:05] FIRING: MySQLReplicaNotUsingGTID: MySQL replica db2252:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID [05:44:01] FIRING: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db2143:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:48:43] all these are expected, downtime expired [05:52:05] RESOLVED: MySQLReplicaNotUsingGTID: MySQL replica db2252:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID [13:00:34] backups are working ok on db2250, so we will likely decom db2141 on tuesday [13:06:06] Hi folks - I'm going to be doing the essential work summary for the week about 15:00 UTC (so about 2 hours from now), so please update the gdoc before then :)