[14:43:43] <cdanis>	 volans: do we expect the OOMs on wt-static to stop as well?
[14:44:29] <volans>	 cdanis: that's a good question, but my hope is yes, those might have been related to the full disk and broken import and such
[14:45:15] <volans>	 what I was not able to understand yesterday night was why python-requests was not there, and apt logs didn't help, but it might have happened before the last log rotation
[14:45:35] <volans>	 also there is unattended upgrades there, so maybe was removed by autoremove, dunno
[14:45:45] <cdanis>	 i don't believe unattended-upgrade does autoremove
[14:46:07] <volans>	 I didn't set it up but I was told it used to work just fine
[14:46:30] <volans>	 oh I wasn't implying that one runs the other
[14:47:00] <cdanis>	 oh okay
[14:47:29] <moritzm>	 unattended-upgrades has an option for autoremove, but not sure enabled by default or not
[14:48:04] <volans>	 cdanis: I've subscribed you to the task apergos has opened today
[14:48:10] <volans>	 after the investigation
[14:48:28] <volans>	 there are both 01autoremove and 01autoremove-kernels in apt.conf.d/
[14:48:38] <volans>	 in addition to the unattended-upgrades ones
[14:49:49] <volans>	 cdanis: and from my estimations we have another ~12GB that could be cleared if we manage to follow suggestions in that task
[14:49:57] <volans>	 so not sure the double disk space is actually needed
[14:50:00] <cdanis>	 yeah I saw your post re: other unreferenced images
[16:01:15] <cdanis>	 lol, this is atlas_exporter's metrics output after running on a single measurement for a few minutes: https://phabricator.wikimedia.org/P9807
[16:02:52] <godog>	 that's very detailed for sure!
[16:02:57] <cdanis>	 I think too much
[16:03:24] <cdanis>	 in addition to wishing that the latency metrics indicated milliseconds, I want to drop a few of these labels, I want to aggregate others
[16:04:03] * herron resists urge to quote “I think too much”
[16:04:09] <cdanis>	 lol
[16:22:03] <jynus>	 I don't think it is very important
[16:22:16] <jynus>	 but every time I run puppet on prometheus hosts I get
[16:22:19] <jynus>	 Notice: /Stage[main]/Rsync::Server/Service[rsync]/ensure: ensure changed 'stopped' to 'running'
[16:22:30] <jynus>	 and if I do "systemctl status rsync"
[16:22:42] <jynus>	 Active: inactive (dead) ConditionPathExists=/etc/rsyncd.conf was not met
[16:23:01] <jynus>	 let me know if you want me to file a ticket or I should ignore it or anything
[16:24:28] <moritzm>	 that was maybe a left over of some prometheus data migration? godog might know
[16:25:44] <godog>	 jynus: yeah a task would be great, thank you!
[16:25:51] <jynus>	 so I have no problem and it is not blocking me, it was just a FYI
[16:25:51] <godog>	 yeah I think it is what moritzm said
[16:25:55] <jynus>	 I will then file it
[16:28:07] <jynus>	 https://phabricator.wikimedia.org/T239837
[16:28:19] <jynus>	 not sure what are good tags of it, only added the one
[16:28:26] <jynus>	 thanks, cheers
[16:28:33] <godog>	 jynus: thanks! yeah observability tag works