[00:36:56] <andrewbogott>	 I quelled the toolsbeta alerts but now they're firing again and I can't get excited about investigating on a Friday night so I'm just silencing things for now.
[16:40:59] <dcaro>	 it seems toolsbeta prometheus is out of space :/ 'log samples: write /srv/prometheus/tools/metrics/wal/00000511: no space left on device"'
[16:47:55] <dcaro>	 extended the volume to 20G for now, it's back online :)
[16:50:45] <dcaro>	 that volume is not in tofu though :/, there's one for toolforge-prometheus-1, but not for toolforge-prometheus-2
[16:51:45] <dcaro>	 there's something weird going on too ,the toolsbeta-prometheus-1 volume seems to be attached to an instance (just by id), that does not really exist
[16:53:37] <dcaro>	 created T396276 to follow up
[16:53:37] <stashbot>	 T396276: [toolsbeta,tofu,infra] There's some discrepancy between the volumes in toolsbeta and tofu - https://phabricator.wikimedia.org/T396276