Automatic retries of hooks

[Gabriel Samfira]

On Mi, 2016-01-20 at 10:39 -0500, Aaron Bentley wrote:
> On 2016-01-20 10:30 AM, Gabriel Samfira wrote:
> > The auto-retry thing was created to overcome situations in which
> > the machine is rebooted, or chashes during a hook run
> > (independently of juju). In this case, the charm would not be able
> > to recover automatically from a transient situation.
> 
> If the intent was to handle reboots, couldn't it be written to
> restart
> any pending hooks after a reboot, rather than when the hooks fail?

The original intent was to re-run a hook in case of external
intervention outside of juju. This includes but is not limited to:

* automatic reboots
* OOM situation
* power outage
* killall -9 jujud (chaos monkey/gremlins/postal sysadmin)

This has grown to automatically retry on any failure. While retrying
 once at agent startup is enough for *some* needs, it may not be enough
for other charms. I would not remove the current behavior. I would
simply make it configurable in case the current behavior does not suit
everyone. The auto retry on all errors is a safe bet for charms that do
not implement retry logic, and as William stated, retrying an operation
inside a hook, will block all other hooks form running.

Just my 2 cents.

> Even re-running just at agent-startup would be a lot clearer.
> 
> Aaron