[06:47:49] federico3: are you the best person to handle the "Users with empty passwords detected" mail? [07:07:26] no but I can try [07:14:36] good luck, then :) [or we can find someone else to ask] [12:03:34] switching over x3 in eqiad [12:12:25] FIRING: SystemdUnitFailed: swift_dispersion_stats_lowlatency.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:17:25] FIRING: [2x] SystemdUnitFailed: swift_dispersion_stats_lowlatency.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:22:25] FIRING: [3x] SystemdUnitFailed: swift_dispersion_stats.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:27:25] FIRING: [3x] SystemdUnitFailed: swift_dispersion_stats.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:32:25] FIRING: [3x] SystemdUnitFailed: swift_dispersion_stats.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:51:08] I expect those to resolve now swift/thumbor are happy again; if not I'll kick them in a bit [13:12:25] RESOLVED: SystemdUnitFailed: swift_ring_manager.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:09:27] federico3: the afl schema change, don't start a new section. Since the whole dc is depooled, I can go a bit crazy [14:09:37] I do it with live=True [14:10:21] it's currenly waiting for the next round [14:10:47] just ctrl + c it for now [14:11:08] it's gonna take forever [14:11:41] I'll let it finish the current section [14:12:08] I mean it's currently repooling db1259, want me to stop it? [14:12:29] yeah [14:12:32] well worst case I repool that by hand [14:12:47] yup but I can do it [14:12:51] ok [14:13:04] then I just stopped the script [14:41:14] Running the schema changes now [14:41:25] it's so nice being able to do it so quickly [14:51:22] heh