Intentionally introducing failures into Juju

[Stuart Bishop]

On 14 August 2014 20:39, Gustavo Niemeyer
<gustavo.niemeyer at canonical.com> wrote:
> On Thu, Aug 14, 2014 at 3:42 AM, Stuart Bishop
> <stuart.bishop at canonical.com> wrote:
>> Further to just injecting failures, I'm interested in controlling when
>> and the order hooks can run. A sort of manual mode, which could be
>> driven by a test harness such as Amulet.
>
> This sounds quite heavyweight and intrusive. Introducing delays in
> certain actions so that the flow is altered sounds okay, but manually
> modifying the order of hooks defined to be the correct one doesn't
> sound right. The former is also more useful in stress testing, while
> the latter is going to be seldom used because people need to think
> through the cases that could explode, and then hand-code the failure
> scenario.

I think tests requiring particular hook orderings would be more common
than you believe. Stress testing, and integration testing in general,
often trip over these bugs. At the moment, we just fix them. I'd like
to be able to fix them, and in addition add tests so it doesn't break
that way again. I think it will encourage robust code that works 100%
of the time, rather than the current more common approach of working
most of the time.

Do you mean heavy and intrusive in juju-core, or at the client end? I
would only expect this to be used by test harnesses so am not
concerned about the difficulty or performance to clients.

If it is too intrusive in juju-core, I think I can implement enough of
what I need in charm-helpers (and all of it if we get a generic 'hook
retry' mechanism).


>> Perhaps all hooks in the
>> queue are initially held, and I can unhold them one at a time.
>
> You should be able to do that one with debug-hooks today already, right?

Yes, and if I put checks for certain lock files in the charm-helpers
@hook decorator, I can get similar behaviour. If I had a 'hook retry'
I could even get arbitrary ordering. Blocking is enough for now, as I
don't think I currently have use cases for arbitrary ordering.


-- 
Stuart Bishop <stuart.bishop at canonical.com>