[02:11:34] FIRING: DiskSpace: Disk space thanos-be2008:9100:/srv/swift-storage/accounts0 0.07657% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:11:34] FIRING: DiskSpace: Disk space thanos-be2008:9100:/srv/swift-storage/accounts0 0.07517% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:33:28] That's going to be T443888 again :-/ [08:33:28] T443888: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888 [10:11:35] FIRING: DiskSpace: Disk space thanos-be2008:9100:/srv/swift-storage/accounts0 0.07346% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:21:35] RESOLVED: DiskSpace: Disk space thanos-be2008:9100:/srv/swift-storage/accounts0 0.0734% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=thanos-be2008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:55:39] damn it, thanos-swift is totally failing to replicate the tegola container db (which is huge) cross-DC [15:57:24] Emperor: do you know the cause? [15:58:43] I'm guessing the huge size isn't helping. I've spent all day on digging so far, tried upping node_timeout, not getting very far. It rsyncs the db file across and then complains e.g. container-replicator: ERROR reading HTTP response from {'device': 'accounts1', 'id': 1, 'ip': '10.64.132.16', 'meta': '', 'port': 6001, 'region': 1, 'replication_ip': '10.64.132.16', 'replication_port': 6001, 'weight': 100.0, 'zone': 7, 'index': 1}: [15:58:43] RemoteDisconnected('Remote end closed connection without response') [15:59:01] Which is, I think, that the remote end is failing to do the newid-and-move-into-place [15:59:26] I'm currently looking at driving that process by hand, which is picking up that sometimes rsync is copying over a file that isn't in fact a valid sqlite3 file [16:00:26] After I VACUUMed the file it's "only" 18G rather than 27G, but this is still huge (much larger than the problematic thumbnail container dbs) [16:01:56] _perhaps_ issue is that the file is so huge and currently being updated quite rapidly, and this means that rsync has a low chance of transferring a valid DB? When I run an integrity check on the source end it's looked good each time. [16:02:32] Is it trying to do a delta transfer (as you said it isn't valid)? [16:03:23] No, it does a --whole-file by default (and I've seen rsync thus running) [16:03:29] +1 [16:04:27] I'm currently checking the dropped files left in the tmpdir from earlier. [16:04:38] [documenting in T443888 as I go] [16:04:39] T443888: accounts0 full on thanos-be2008 - https://phabricator.wikimedia.org/T443888 [16:21:54] https://sqlite.org/rsync.html and https://til.simonwillison.net/sqlite/compile-sqlite3-rsync might be of interest [16:24:24] Mmm, but it needs not to be a byte-for-byte copy in the end, the received makes up a new id and sets up some info about when it was last synced. [16:24:52] And at least some of the tmp entries are valid, so it's not just a case of that failing (otherwise presumably this would happen more often) [16:25:15] [though I guess you don't often expect to use the rsync-update process] [16:27:27] ...and I agree swift upstream probably wants to use it. [16:33:53] Yep, given that it documented then it might be a good idea, even if via an option. That is me out of ideas though amd I had better head off to collect my daughter from the station. See you tomorrow sometime, smooth travels! [17:04:15] OK, manually doing the post-rsync stuff on thanos-be1005 looks to have worked. [17:04:27] I'll try similar on thanos-be1006 after a typo break [17:54:38] container-replicator: 5196 successes, 0 failures [17:54:39] yesss [19:01:04] hopefully that's thanos-swift back into a plausible state before I leave the country tomorrow :-/