[Maas-devel] RFC: "Serialising" power actions

[Gavin Panella]

On 23 September 2014 11:02, Mark Shuttleworth <mark at ubuntu.com> wrote:
> On 17/09/14 09:58, Gavin Panella wrote:
>>
>>>  * Storing state in the pserv without a means to recover it is a
>>> recipe for disaster
>> I guess you mean that a crash or restart in pserv would mean that
>> in-progress power commands wouldn't be resumed. That's true, but it's
>> not a disaster. It means that for nodes in all states but DEPLOYED we
>> need to wait for the periodic power monitor to notice and reissue a
>> command (see later; it doesn't do this yet). For DEPLOYED nodes, sure,
>> the command will currently be lost, but these nodes are, one assumes,
>> under active management, and some process outside of MAAS will notice,
>> be that a human or a Juju or something else.
>
> So (a) guess

Not really. For states other than deployed we can take a stance on what
power state the node ought to be in, and get MAAS to converge on that.

> and (b) hope someone else cleans up the problem?

It's not ideal, but it's better than what we had before, where MAAS did
remember all outstanding power commands issued (unless RabbitMQ broke),
but then ran them concurrently, and didn't give any feedback.

It's not a /disaster/ because we're not going to be restarting cluster
controllers frequently, and crashes too will hopefully be infrequent.
Losing in-progress power changes is a relatively small problem compared
to the above.

Solving this isn't a code problem, it's about the behaviour we'd want:
restarting in-progress commands when a cluster controller comes back up
/might/ be the wrong thing to do.

For example, given notice of an imminent power outage, I send a
power-off command to all my nodes. The power fails prematurely (or I was
late issuing the command) and my whole cluster goes suddenly dark. When
the power is restored the cluster controller needs to do a lengthy fsck,
or just boots slowly. The nodes in my cluster are okay, and they boot as
soon as power returns, if set in the BIOS, or I switch them on by hand,
so that service is restored for my customers. A few minutes later the
cluster controller finishes booting and it resumes all in-progress power
commands, turning all my nodes off.

Right now I'm not sure if we can completely codify what to do after an
outage or crash. We might be able to address the hypothetical situation
above by putting an expiry time on each power command, but for how long
should that be? That would need discussion and/or experimentation.

Perhaps the next thing /is/ to blindly resume in-progress commands, then
we can refine iteratively from there.