[06:55:51] circular replication has been disabled everywhere [06:55:54] see you all Wed! [06:57:49] This also happened: https://phabricator.wikimedia.org/T442800#12487833 [08:18:42] :sadpanda: [09:34:20] You can be even more sad now: https://phabricator.wikimedia.org/T442815 [09:42:28] :( [10:11:37] At least the the first one came back, the latest one shows "System BIOS has halted" [13:36:48] FIRING: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db1228:9104 has too large replication lag (1h 25m 19s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1228&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [13:37:48] FIRING: MysqlReplicationLag: MySQL instance db1228:9104@s4 has too large replication lag (1h 2m 21s). Its replication source is db1160.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1228&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [13:41:29] ^ recovering from earlier [13:41:48] RESOLVED: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db1228:9104 has too large replication lag (4m 18s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1228&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [13:42:48] RESOLVED: MysqlReplicationLag: MySQL instance db1228:9104@s4 has too large replication lag (4m 18s). Its replication source is db1160.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1228&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [19:24:40] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1228:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:57:39] FIRING: MySQLDepooledNotSilenced: MySQL instance pc1017 pc7 depooled but not silenced - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/fc7sbw8/mariadb-pooling-and-silences-overview - https://alerts.wikimedia.org/?q=alertname%3DMySQLDepooledNotSilenced [21:09:40] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1228:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:32:39] RESOLVED: MySQLDepooledNotSilenced: MySQL instance pc1017 pc7 depooled but not silenced - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/fc7sbw8/mariadb-pooling-and-silences-overview - https://alerts.wikimedia.org/?q=alertname%3DMySQLDepooledNotSilenced