[11:29:36] just a note on the thumbs stuff - if you'd like to speed things up a little and are concerned about thumbor, it appears to be weathering this quite well. Some scrapers were creating a lot of noise and load when we first started this but they have been either fended off or finished scraping and thumbor is all quiet [11:50:25] FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:53:57] hnowlan: Thanks! We are almost there anyway, 90%. It'll be 100% at Wednesday. If you're talking about the one-off clean up. I highly highly doubt with the rate we are deleting it could put pressure on thumbor. It's millions per day but still in terms of portion of thumbnails being removed it's peanuts (1/1000th per day). Making it 1/500th won't make a dent [11:55:10] rclone> lost a couple of deletion races [11:55:54] I'm going to try running the mw script for categorylinks again (on s2). Last time, it caused a lot of lag but zabe fixed it. Heads up just in case. [11:57:00] grand, thanks a lot for the work! [11:58:39] RESOLVED: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:31:23] we'll probably need to do ms-be1065 and ms-be1066 soon. at 93% https://grafana.wikimedia.org/d/000000378/ladsgroup-test?forceLogin&from=now-30d&orgId=1&to=now&viewPanel=28 [12:31:44] I'm waiting for the deletion to remove enough so the drop would be bigger [12:55:29] Yeah, I'm generally content to act once they start appearing on the alerts page for now [13:01:12] <_joe_> o/ I'm installing a new version of conftool on the puppet servers and then on the cumin hosts; given there's a change to dbctl (which should be a noop), please let me know if anything fails [13:01:30] _joe_: thanks for the heads up [13:20:11] FYI, started the file data migration in s4