Juju Leader Election and Application-Specific Leadership

[Dmitrii Shcherbakov]

I think there should be two different terms explicitly mentioned, e.g.
a 'juju leader unit' and an 'application master'. Plus I would add
some discussion around a lead unit being juju-specific and rather
being a single point control for operational code and not related at
all to application-specific leadership. Simply put: "if your
application supports designating a master manually it can be done from
a leader unit - not the other way around."

It is easy to get lost trying to wrap your head around the that fact
and think of a wrong design knowing that it might be possible to force
a leader unit election.

----

Another thing in the context of leadership:

https://jujucharms.com/docs/2.0/authors-charm-leadership#leadership-tools

"Every service deployed by juju also has access to a
***pseudo-relation*** over which leader settings can be communicated
with the following tools:"

The term "pseudo-relation" is kind of vague. It almost makes you think
there is a (peer) relation context during the
'leader-settings-changed' hook execution (or, similarly, for
leader-elected and leader-deposed events).

If you follow this then you will realize that there's no relation context:

ContextLeadership in jujuc:
https://github.com/juju/juju/blob/staging/worker/uniter/runner/jujuc/context.go#L172

LeadershipContext:
https://github.com/juju/juju/blob/staging/worker/uniter/runner/context/leader.go#L41

coreContext which uses newLeadershipContext
https://github.com/juju/juju/blob/staging/worker/uniter/runner/context/contextfactory.go#L140

HookContext uses coreContext and checks for hookInfo.Kind.IsRelation()
before populating the relation info:
https://github.com/juju/juju/blob/staging/worker/uniter/runner/context/contextfactory.go#L184

In the 'charm' repo (not juju repo) IsRelation is defined
(LeaderSettingsChanged kind is not considered a "relation hook") hence
no relation context will be added to the hook execution.
https://github.com/juju/charm/blob/v5/hooks/hooks.go#L96

// IsRelation returns whether the Kind represents a relation hook.
func (kind Kind) IsRelation() bool {
    switch kind {
        case RelationJoined, RelationChanged, RelationDeparted, RelationBroken:
            return true
    }
    return false
}

In fact, I think that leader unit election will be performed even
without a peer relation (when you only have one unit in an
application) - it would make total sense to do that because in general
there can be any number of units (including one) and the charm logic
should not break in this case, meaning that there should be a leader.

The reason this is important to mention is that when you write a
reactive handler you need to know if there is going to be a relation
argument or not. If you don't have it but expect it there will be an
error in your code. Or, if you use a default argument, there will be a
piece of dead code in a handler:

@when(leadership.is_leader)
def handler(rel)  # <--- error
    rel.set_remote('x', 'y')

or:

@when(leadership.is_leader)
def handler(rel=None)
    if rel:
        rel.set_remote('x', 'y') # <--- dead code


Thoughts?
Best Regards,
Dmitrii Shcherbakov


Dmitrii Shcherbakov  | Canonical
Field Software Engineer
dmitrii.shcherbakov at canonical.com
IRC (freenode): Dmitrii-Sh


On Thu, Apr 6, 2017 at 10:51 AM, Stuart Bishop
<stuart.bishop at canonical.com> wrote:
> On 6 April 2017 at 00:26, Dmitrii Shcherbakov
> <dmitrii.shcherbakov at canonical.com> wrote:
>
>> https://jujucharms.com/docs/2.1/reference-charm-hooks#leader-elected
>> "leader-elected is run at least once to signify that Juju decided this
>> unit is the leader. Authors can use this hook to take action if their
>> protocols for leadership, consensus, raft, or quorum require one unit
>> to assert leadership. If the election process is done internally to
>> the service, other code should be used to signal the leader to Juju.
>> For more information read the charm leadership document."
>>
>> This doc says
>> "If the election process is done internally to the service, other code
>> should be used to signal the leader to Juju.".
>>
>> However, I don't see any hook tools to assert leadership to Juju from
>> a charm based upon application-specific leadership information
>> http://paste.ubuntu.com/24319908/
>>
>> So, as far as I understand, there is no manual way to designate a
>> leader and the doc is wrong.
>>
>> Does anyone know if it is supposed to be that way and if this has not
>> been implemented for a reason?
>
> I agree with your reading, and think the documentation is wrong.  If
> the election process is done internally to the service, there is no
> way (and no need) to signal the internal 'leader' to Juju.
>
> I also put 'leader' in quotes because if your service maintains its
> own master, you should not call it 'leader' to avoid confusion with
> the Juju leader.
>
> For example, the lead unit in a PostgreSQL service appoints one of the
> units as master. The master remains the master until the operator runs
> the 'switchover' action on the lead unit, or the master unit is
> destroyed causing the lead unit to start the failover process. At no
> point does Juju care which unit is 'master'. Its communicated to the
> end user using the workload status. Its simple enough to do and works
> well.
>
>
> --
> Stuart Bishop <stuart.bishop at canonical.com>