Automatic retries of hooks

[Gabriel Samfira]

The auto-retry thing was created to overcome situations in which the machine is rebooted, or chashes during a hook run (independently of juju). In this case, the charm would not be able to recover automatically from a transient situation.

This scenario is more evident in Windows workloads, where some features like Hyper-V require a reboot. This is fine, because you can install the feature with a -NoReboot flag, and use the juju-reboot --now tool to safely reboot.

After the machine comes back up, Windows still needs to configure  the new feature. While it is configuring the feature, the services start (including juju), and hook execution starts up (as its supposed to).   The problem is that as part of the feature configuration, the system needs to reboot one final time (Windows....right?). This is done automatically by the feature installer, outside of juju.

This causes the hook to error, but not because of an actual problem. For this scenario, its enough to retry once when the unit agent comes up.

A solution might be to make this feature configurable. Something like retry profiles:

* periodic (current behavior)
* one-shot (once at agent startup)
* disabled

Gabriel

On Mi, 2016-01-20 at 09:39 -0500, Charles Butler wrote:
I'm pretty sure that we have amenities to reboot the host without completely skewing the hook execution

https://jujucharms.com/docs/1.25/reference-hook-tools#juju-reboot-[--now]

This should have rebooted the machine after safely closing out of any hook context the charm was in, and upon reboot it should have resumed from the next context in queue.  I'm not a huge fan of a charm doing auto hook retries, for the reasons outlined by Rick, unless it is well understood and documented behavior.  Just chiming in with my 2 cents


Charles Butler <charles.butler at canonical.com<mailto:charles.butler at canonical.com>> - Juju Charmer
Come see the future of datacenter orchestration: http://jujucharms.com

On Wed, Jan 20, 2016 at 9:22 AM, Rick Harding <rick.harding at canonical.com<mailto:rick.harding at canonical.com>> wrote:
+1 retries are great, with backoff, when you know you're doing it because you have experience that certain api requests to clouds, or to other known failure points.

Blindly just saying "if at first you don't succeed, go go go" isn't a better UX. It adds another layer of complexity in debugging, and doesn't really improve the product. Only the charm author knows enough about what it's trying to achieve to do intelligent retry.

In this case, if there's something about unexpected reboots of machines, perhaps there's some specific case that Juju can grow some intelligence and hint at the charm author what happened. The charm can then react to that information as it deems necessary.

On Wed, Jan 20, 2016 at 8:42 AM Dean Henrichsmeyer <dean at canonical.com<mailto:dean at canonical.com>> wrote:
Hi,

It seems the original point James was making is getting missed. No one is arguing over the value of being able to retry and/or idempotent hooks. Yes, you should be able to retry them and yes nothing should break if you run them over and over.

The point made is that Juju shouldn't be automatically retrying them. The argument of "no one knows what went wrong so Juju automatically retrying them is a better experience" doesn't work. The intelligence of the stack in question, regardless of what it is, goes in the charms. If you start conflating and mixing up where the intelligence goes then creating, running, and debugging those distributed systems will be a nightmare.

The magic should only be in Juju's ability to effectively drive the models and intelligence encoded in the charms. It shouldn't make assumptions about what that intelligence is or what those models require.

Thanks.


-Dean
--
Juju-dev mailing list
Juju-dev at lists.ubuntu.com<mailto:Juju-dev at lists.ubuntu.com>
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev


--
Juju-dev mailing list
Juju-dev at lists.ubuntu.com<mailto:Juju-dev at lists.ubuntu.com>
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev



--
Juju-dev mailing list
Juju-dev at lists.ubuntu.com<mailto:Juju-dev at lists.ubuntu.com>
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160120/0e10b9c4/attachment-0001.html>