Controllers running out of disk space

[John Meinel]

So logs in mongo and logs on disk should be capped, and purged when they
get above a certain size. 'audit.log' should never be automatically purged.
Charms in the blobstore are potentially local data that we can't reproduce,
so hard to automatically purge them. I think there has been some work done
to properly reference count them, so we at least know if anything is
currently referencing them. And we could purge things that are from a
charmstore, since we know it is available elsewhere.

I'm fine with a "pause" sort of mode so that you have an opportunity to
move things out, and some sort of manually triggered garbage collection.
But if we do know that some garbage collection is safe, we should probably
just do it by default.

John
=:->


On Thu, Nov 17, 2016 at 12:20 PM, Uros Jovanovic <
uros.jovanovic at canonical.com> wrote:

> Hi all,
>
> I'd like to start a discussion on how to handle storage issues on
> controller machines, especially what we can do when storage is getting 95%
> or even 98% full. There are many processes that are storing data, we have
> at least:
> - charms and resources in the blobstore
> - logs in mongo
> - logs on disk
> - audit.log
>
> Even if we put all models in the controller into read-only mode to prevent
> upload of new charms or resources to blobstore, we still have to deal with
> logs which can fill the space quickly.
>
> While discussing about this with Rick, given the work on model migrations,
> maybe the path forward would be to allow admin to put the whole controller
> in "maintenance" mode and put all agents on "hold".
>
> How to deal with storage issue after that is open to discussion, maybe we
> need tools to "clear" blobstore, or export/compress/truncate logs, etc.
>
> Thoughts?
>
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20161117/17715b99/attachment.html>