Controllers running out of disk space

Nate Finch nate.finch at canonical.com
Thu Nov 17 16:07:08 UTC 2016


Resources are also stored in mongo and can be unlimited in size (not much
different than fat charms, except that at least they're only pulled down on
demand).

We should let admins configure their max log size... our defaults may not
be what they like, but I bet that's not really the issue, since we cap them
smallish (the logs stored in mongo are capped, right?)

Do we still store all logs from all machines on the controller?  Are they
capped?  That has been a problem in the past with large models.

On Thu, Nov 17, 2016 at 8:53 AM Rick Harding <rick.harding at canonical.com>
wrote:

> I'm definitely agreeing we need to provide some better tools for the admin
> of the controller to track and garden things such as charms and resources
> which can be quite large and grows over time.
>
> My main point with Uros was that we have this way due to model migrations
> to stick a controller/model into a quiet mode so that a migration can take
> place. It feels like a natural fit for a bit of a maintenance mode as
> operators manage large scale/long running Juju controllers over time. I
> wanted to look into the feasibility of allowing the operator to engage the
> quiet mode while they manage disk or other issues.
>
> The other question is, is this a temp issue? When you migrate models to
> another controller only the latest charm/resource revision goes with it.
> Maybe there's a place for a migration as part of good hygiene. It feels a
> bit forceful, but actually might be a safe practice.
>
> On Thu, Nov 17, 2016 at 4:13 AM John Meinel <john at arbash-meinel.com>
> wrote:
>
> So logs in mongo and logs on disk should be capped, and purged when they
> get above a certain size. 'audit.log' should never be automatically purged.
> Charms in the blobstore are potentially local data that we can't reproduce,
> so hard to automatically purge them. I think there has been some work done
> to properly reference count them, so we at least know if anything is
> currently referencing them. And we could purge things that are from a
> charmstore, since we know it is available elsewhere.
>
> I'm fine with a "pause" sort of mode so that you have an opportunity to
> move things out, and some sort of manually triggered garbage collection.
> But if we do know that some garbage collection is safe, we should probably
> just do it by default.
>
> John
> =:->
>
>
> On Thu, Nov 17, 2016 at 12:20 PM, Uros Jovanovic <
> uros.jovanovic at canonical.com> wrote:
>
> Hi all,
>
> I'd like to start a discussion on how to handle storage issues on
> controller machines, especially what we can do when storage is getting 95%
> or even 98% full. There are many processes that are storing data, we have
> at least:
> - charms and resources in the blobstore
> - logs in mongo
> - logs on disk
> - audit.log
>
> Even if we put all models in the controller into read-only mode to prevent
> upload of new charms or resources to blobstore, we still have to deal with
> logs which can fill the space quickly.
>
> While discussing about this with Rick, given the work on model migrations,
> maybe the path forward would be to allow admin to put the whole controller
> in "maintenance" mode and put all agents on "hold".
>
> How to deal with storage issue after that is open to discussion, maybe we
> need tools to "clear" blobstore, or export/compress/truncate logs, etc.
>
> Thoughts?
>
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20161117/ca819fc9/attachment.html>


More information about the Juju mailing list