Intentionally introducing failures into Juju

[Gustavo Niemeyer]

On Thu, Aug 14, 2014 at 3:42 AM, Stuart Bishop
<stuart.bishop at canonical.com> wrote:
> Further to just injecting failures, I'm interested in controlling when
> and the order hooks can run. A sort of manual mode, which could be
> driven by a test harness such as Amulet.

This sounds quite heavyweight and intrusive. Introducing delays in
certain actions so that the flow is altered sounds okay, but manually
modifying the order of hooks defined to be the correct one doesn't
sound right. The former is also more useful in stress testing, while
the latter is going to be seldom used because people need to think
through the cases that could explode, and then hand-code the failure
scenario.

> Perhaps all hooks in the
> queue are initially held, and I can unhold them one at a time.

You should be able to do that one with debug-hooks today already, right?

> This would let me test the odd edge cases, such as peers departing
> peer relations during handshaking, or what happens when a new client

That's not just reordering, but introducing a failure scenario where a
peer unit leaves the relation. It would indeed be useful to support
that sort of failure injection, but with a proper mechanism for it
rather than fiddling with hook ordering.

> unit is added and its relation-changed hooks manages to run before the
> relation-joined hooks at the server end.

Similar case.

> If you could do this, you could inject your failures by actually
> breaking your units using juju run or juju ssh. Deploy your units, run
> the install hooks, juju ssh in breaking one of the units (rm -rf /,
> whatever), run the peer relation hooks, confirm that the service is
> still usable despite the failed unit.

Similar case as well. We should be able to inject those failures
without having to manually fiddle with hook ordering.


gustavo @ http://niemeyer.net