Controllers running out of disk space

[Rick Harding]

I'm definitely agreeing we need to provide some better tools for the admin
of the controller to track and garden things such as charms and resources
which can be quite large and grows over time.

My main point with Uros was that we have this way due to model migrations
to stick a controller/model into a quiet mode so that a migration can take
place. It feels like a natural fit for a bit of a maintenance mode as
operators manage large scale/long running Juju controllers over time. I
wanted to look into the feasibility of allowing the operator to engage the
quiet mode while they manage disk or other issues.

The other question is, is this a temp issue? When you migrate models to
another controller only the latest charm/resource revision goes with it.
Maybe there's a place for a migration as part of good hygiene. It feels a
bit forceful, but actually might be a safe practice.

On Thu, Nov 17, 2016 at 4:13 AM John Meinel <john at arbash-meinel.com> wrote:

So logs in mongo and logs on disk should be capped, and purged when they
get above a certain size. 'audit.log' should never be automatically purged.
Charms in the blobstore are potentially local data that we can't reproduce,
so hard to automatically purge them. I think there has been some work done
to properly reference count them, so we at least know if anything is
currently referencing them. And we could purge things that are from a
charmstore, since we know it is available elsewhere.

I'm fine with a "pause" sort of mode so that you have an opportunity to
move things out, and some sort of manually triggered garbage collection.
But if we do know that some garbage collection is safe, we should probably
just do it by default.

John
=:->


On Thu, Nov 17, 2016 at 12:20 PM, Uros Jovanovic <
uros.jovanovic at canonical.com> wrote:

Hi all,

I'd like to start a discussion on how to handle storage issues on
controller machines, especially what we can do when storage is getting 95%
or even 98% full. There are many processes that are storing data, we have
at least:
- charms and resources in the blobstore
- logs in mongo
- logs on disk
- audit.log

Even if we put all models in the controller into read-only mode to prevent
upload of new charms or resources to blobstore, we still have to deal with
logs which can fill the space quickly.

While discussing about this with Rick, given the work on model migrations,
maybe the path forward would be to allow admin to put the whole controller
in "maintenance" mode and put all agents on "hold".

How to deal with storage issue after that is open to discussion, maybe we
need tools to "clear" blobstore, or export/compress/truncate logs, etc.

Thoughts?



--
Juju mailing list
Juju at lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju


--
Juju mailing list
Juju at lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20161117/c11c7f17/attachment.html>