Automatic retries of hooks

[Bogdan Teleaga]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hello everybody,

This has been a WIP for a while now so maybe some of you have heard
about it.

It all started out with us needing to have hook retried after a random
reboot and it evolved into retrying hooks upon any kind of failure.

So as of now failing hooks will be retried automatically after a
certain time. The minimum wait time will be 20 seconds, while the
maximum will be 20 minutes and it's going to increase with a factor of
2 for every failure. Also a small jitter is introduced for a bit of
randomness. Using juju resolved will overwrite this timer and cause it
to restart at the beginning.

I've tested it for a while and it has proven to be relatively robust
in my tests. Probably having a CI test soonish would be recommended.

The waiting amount has been chosen relatively arbitratily so if anyone
has comments or ideas for that, I'm open to suggestions. The
discussion for that should go
here(https://github.com/juju/juju/pull/3835), since apparently I
merged the branch with some values I used in testing and did not
change them back to the intended ones.

Regards,
Bogdan
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJWVwiPAAoJEIeXGztzAn9lzNEQALYGq0hp1PWuWy0iADNVpgaV
akgbnxZgqGGCb7ZRH2Lz/fnJAgTQXUNnmi0fI0wDzRwTbtQI/t2fZt1huKzSjtul
AtPzCVlSg5zsolIfbvp3LvdPoneoKrO0W919KYNIw+yPv8mFstqf8oubCmbCn3iF
NbfZ2OR4FyvM9KNH7PE2f2PGhrb4Q2S5sZcE9L65fxmZTTGiplAUOW3rTe5clrdx
LT3aM6RtoOP0D8CzmX/RZLgp9Vrm1hd/Lju+NSc1aupDwid3B8DIERR0891FgQ9z
KnRs9E4w41TZVV3Z3lXr6r47OV0oPgRbs5nLzD1AcAVwLaKB8K975t4BMwbtGbFd
7gtc9obzwMZJ408HPSvlNetxMHb4UwPFn9I/5+G4vBjhhm+I13RNf/pHMrplThaG
ETD4tSUhklN9ISkVPeXU39oY8i5wJNKV54krW2q9BwIRp4LpZnpgFyXh3wdrxow4
4z6NYQ00k1zSdQ8UY74H9zaww6lJAr+HBRXP/zVPpUvm9xaytFx4KReRSnbNu5Dp
r165CuBysPyT/MnglRNUZT/h3/zVKHTpRruOXRagy/y6g+gcoQi3LGnpAqtxzI1D
BLV+B8FP3SwkSBIeDHPRw2Nyo0mydRqgob+WgaOaVGnxIcoLUTeOa4kXK87I5X97
+nr3BdQbXaGugbEmD9xq
=AwIa
-----END PGP SIGNATURE-----