[00:43:55] just fyi T428832 [00:43:55] T428832: db1262 crashed - https://phabricator.wikimedia.org/T428832 [06:41:35] FIRING: MySQLReplicaNotUsingGTID: MySQL replica db2226:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID [06:42:55] ^ me testing [06:46:35] RESOLVED: MySQLReplicaNotUsingGTID: MySQL replica db2226:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID [10:02:58] Amir1: Due to https://phabricator.wikimedia.org/T428852 I need to depool the only replica left on es5 codfw, so that means leaving the master on its own. Load wise isn't a problem, but is it a problem for MW to have a section with just the master and no replicas? [10:10:37] if that is a big issue, I can always use an eqiad replica to clone the codfw one, but given that it is almost 7tb, i'd rather use the same dc [12:47:10] marostegui: ooo today but theoretically it should be fine as most of vanilla mediawiki setups have a master with load and no replicas [13:57:21] Is there any reason that db2202 doesn't appear to have been pooled? I ran the reimage and once replication had gone green the cookbook ended with a PASS. Scrolling back, no depooling happened [14:18:12] OK, seems to be a test host (test-s1), but unlike test-s4 it shows up in s1 [14:39:02] cezmunsta: yeah, long story short....that's normal. I'll give more context on monday [14:39:24] kk [14:40:00] It's confusing indeed yeah [14:40:20] But test-s1 host replicates from s1 but test-s4 is an independent section [14:40:31] For historical reasons I can give on Monday meeting :) [16:04:38] FIRING: SystemdUnitFailed: rsync.service on ms-be2062:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:14:38] RESOLVED: SystemdUnitFailed: rsync.service on ms-be2062:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:19:38] FIRING: [2x] SystemdUnitFailed: rsync.service on ms-be2062:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:27:33] RESOLVED: [2x] SystemdUnitFailed: rsync.service on ms-be2062:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed