[07:01:09] <dcausse>	 o/
[08:49:39] <atsukoito>	 hello!
[09:03:19] <dcausse>	 hi!
[12:55:09] <inflatador>	 <o/
[12:56:29] <ebernhardson>	 \o
[13:01:05] <dcausse>	 o/
[14:33:19] <inflatador>	 dcausse created T432249 for the thanos-swift stuff T432249
[14:33:20] <stashbot>	 T432249: Cirrus Streaming Updater: alert on thanos-swift storage usage/increase observability - https://phabricator.wikimedia.org/T432249
[14:33:25] <dcausse>	 thanks!
[14:34:14] <dcausse>	 we set kubernetes.operator.savepoint.history.max.age: 24h but it's clearly not working :(
[14:35:09] <dcausse>	 all savepoints since ~2024 are retained and we take them every 5min :/
[14:35:23] <ebernhardson>	 !!!
[14:35:41] <dcausse>	 it's around 15Tb...
[14:36:19] <dcausse>	 might disable automatic savepoints if I don't find what's not working
[14:36:42] <ebernhardson>	 is it still the lack of an auto-delete daemon/
[14:38:08] <dcausse>	 this data is not critical
[14:38:08] <ebernhardson>	 i dont remember why, but I have a vague memory sre never setup the daemon that does the TTL cleanups. I had asked about it at some point but they didn't want to run it
[14:38:08] <dcausse>	 ebernhardson: the swift auto-delete daemon is I think something else, it made the problem worse
[14:38:08] <ebernhardson>	 ahh
[14:54:42] <dcausse>	 will try this: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1311066
[14:54:53] <dcausse>	 if it does not work I'll disable these automatic savepoints
[14:57:09] <ebernhardson>	 seems plausible
[14:59:14] <ebernhardson>	 i don't entirely follow the dispose-on-delete doc, "disposed when the generated Kubernetes resource is deleted", which suggests deletes when the pod deletes
[14:59:16] <ebernhardson>	 ?
[14:59:25] <ebernhardson>	 or maybe not pod, but not sure which resourrce
[15:00:55] <dcausse>	 yes I'm confused as well but the doc on "kubernetes.operator.savepoint.cleanup.enabled" says: "For automatic savepoints this can be configured via the kubernetes.operator.savepoint.dispose-on-delete config option"