[Maas-devel] RFC: "Serialising" power actions

[Gavin Panella]

On 17 September 2014 00:43, Julian Edwards <julian.edwards at canonical.com> wrote:
> On Tuesday 16 Sep 2014 11:42:48 Gavin Panella wrote:
>> On 15 September 2014 22:34, Graham Binns wrote:
>> ...
>>
>> > 1: The current power action blocks all others until it as
>> > completed. Other power actions will be queued and executed in turn.
>> > - or -
>> > 2: Each power action supersedes any action that is currently
>> > executing — the existing action is cancelled and then the new
>> > action is run.
>> > - or -
>> > 3. We track the current ("now") and "next" actions for the node,
>> > but drop every action that comes in once those two slots are full.
>>
>> I think #1 is wrong; apart from stress-testing I can't think of a
>> situation where I'd want every panic-and-frustration-induced click of
>> the power buttons in the UI to be recorded and acted upon. I just
>> want it to do the last one.
>>
>> #3 is like #2, but you have to wait for the currently executing
>> command to finish. Boring!
>>
>> I think #2 is the right starting point. In addition:
>
> I disagree. You'll be back in the scenario where important power ops
> are ignored (and is a bug that I just fixed).
>
> For example, a machine is slow to power down and doesn't finish before
> a power up is issued. MAAS expects the node to have rebooted, but in
> fact it hasn't and at best will sit there not doing anything, at worst
> will reboot with old data.
>
> Effectively you've issued two power commands both of which were
> ignored.
>
> We *must* wait for an outstanding power op to finish one way or
> another, whether it fails or it succeeds.

That's a good point. However, a reboot can be modelled as a single power
"task". If a power-off is issued subsequently, then I think it should be
allowed to override the reboot, and transition the machine to a
powered-off state. If a power-on is issued, we can say that it's not
permitted to override the in-progress power task. In other words, where
the final state of an in-progress power task is the same as a subsequent
power task, the subsequent power task should be discarded.

>
>> - If a power-on command is sent to a cluster, and the cluster is
>>   already attempting to turn the node on, the command should be
>>   silently merged with the existing command.
>>
>> - Likewise for power-off.
>
> *Maybe*, I don't think it should be silent, it should respond with
> something to indicate "ok I am already doing that, please wait".
> Whether it's an error condition or not is arguable both ways.

That makes sense, but I definitely don't think it's an error.

>
>> - It may be interesting to include a discriminator from a
>>   monotonically increasing sequence with each power command. A
>>   power-off command that is received by a cluster with a
>>   discriminator lower than a running power-on command would be
>>   rejected.
>>
>>   In practice I doubt this will make much difference, but it's worth
>>   mentioning even if only to reject the idea.
>
> I'm not sure what it achieves tbh, but I'm open to it if you convince
> me :)

Ach, it's British Code, ensuring that first in the queue is always first
in the queue, what.

>
>>
>> > At first glance the second option is simpler — just cancel
>> > whatever's there and then do our thing. But I think that it's
>> > actually a bit
>> >
>> > deceptive. Consider:
>> >  - How do we "cancel" an action?
>>
>> Cancelling means keeping a reference to a cancel function in a shared
>> location in the cluster. Shared state can make people feel dirty, but
>> Twisted's single-thread model makes it pretty benign.
>
> I don't think cancelling is something worth doing, or even possible. I
> am yet to see any power controllers that allow you to cancel an
> in-progress power operation.

Cancellation is possible. For example, an IPMI power-on command looks
something like the following:

    send an IPMI command to power on the node
    wait a bit
    is it up? Yes -> we're done
    send an IPMI command to power on the node
    wait a bit
    is it up? Yes -> we're done
    send an IPMI command to power on the node
    wait a bit
    is it up? Yes -> we're done; no -> that's an error

There are many opportunities to stop in the process above.

>
> The only time you'd be able cancel one is if you queued it up, and I
> already explained why I think that's a bad idea.
>
>> Doing this would also allow the region to ask the cluster things like
>> "are you changing a node's power state?".
>
> The region should know this already if it has an outstanding power
> operation that awaits a response.
>
>>
>> >  - How do we ensure that we're not going to end up in an
>> >  inconsistent state
>> >
>> > if the node is already responding to action #1 when we cancel it?
>>
>> The power control code has been improved to be less fire-and-forget
>> and more fire-and-keep-firing-until-it-is-in-the-desired-state.
>> Cancelling a power-on and starting a power-off should Just Work.
>
> Again, we must wait for these to complete as they are not cancellable
> and can fail, which must result in a node failure.
>
> J