High Availability command line interface - future plans.

[Kapil Thangavelu]

On Thu, Nov 7, 2013 at 2:49 AM, roger peppe <rogpeppe at gmail.com> wrote:

> The current plan is to have a single "juju ensure-ha-state" juju
> command. This would create new state server machines if there are less
> than the required number (currently 3).
>
> Taking that as given, I'm wondering what we should do
> in the future, when users require more than a single
> big On switch for HA.
>
> How does the user:
>
> a) know about the HA machines so the costs of HA are not hidden, and that
> the implications of particular machine failures are clear?
>
> b) fix the system when a machine dies?
>
> c) scale up the system to x thousand nodes?
>
> d) scale down the system?
>
For a), we could tag a machine in the status as a "state server", and
> hope that the user knows what that means.
>
> For b) the suggestion is that the user notice that a state server machine
> is non-responsive (as marked in status) and runs destroy-machine on it,
> which will notice that it's a state server machine and automatically
> start another one to replace it. Destroy-machine would refuse to work
> on a state server machine that seems to be alive.
>
> For c) we could add a flag to ensure-ha-state suggesting a desired number
> of state-server nodes.
>
> I'm not sure what the suggestion is for d) given that we refuse to
> destroy live state-server machines.
>
> Although ensure-ha-state might be a fine way to turn
> on HA initially I'm not entirely happy with expanding it to cover
> all the above cases. It seems to me like we're going
> to create a leaky abstraction that purports to be magic ("just wave the
> HA wand!") and ends up being limiting, and in some cases confusing
> ("Huh? I asked to destroy that machine and there's another one
> just been created")
>
> I believe that any user that's using HA will need to understand that
> some machines are running state servers, and when things fail, they
> will need to manage those machines individually (for example by calling
> destroy-machine).
>
> I also think that the solution to c) is limiting, because there is
> actually no such thing as a "state server" - we have at least three
> independently scalable juju components (the database servers (mongodb),
> the API servers and the environment managers) with different scaling
> characteristics. I believe that in any sufficiently large environment,
> the user will not want to scale all of those at the same rate. For example
> MongoDB will allow at most 12 members of a replica set, but a caching API
> server could potentially usefully scale up much higher than that. We could
> add more flags to ensure-ha-state (e.g.--state-server-count) but we then
> we'd lack the capability to suggest which might be grouped with which.
>
> PROPOSAL
>
> My suggestion is that we go for a "slightly less magic" approach.
> that provides the user with the tools to manage
> their own high availability set up, adding appropriate automation in time.
>
> I suggest that we let the user know that machines can run as juju server
> nodes, and provide them with the capability to *choose* which machines
> will run as server nodes and which can host units - that is, what *jobs*
> a machine will run.
>
> Here's a possible proposal:
>
> We already have an "add-machine" command. We'd add a "--jobs" flag
> to allow the user to specify the jobs that the new machine(s) will
> run. Initially we might have just two jobs, "manager" and "unit"
> - the machine can either host service units, or it can manage the
> juju environment (including running the state server database),
> or both. In time we could add finer levels of granularity to allow
> separate scalability of juju server components, without losing backwards
> compatibility.
>
> If the new machine is marked as a "manager", it would run a mongo
> replica set peer. This *would* mean that it would be possible to have
> an even number of mongo peers, with the potential for a split vote
> if the nodes were partitioned evenly, and resulting database stasis.
> I don't *think* that would actually be a severe problem in practice.
> We would make juju status point out the potential problem very clearly,
> just as it should point out the potential problem if one of an existing
> odd-sized replica set dies. The potential problems are the same in both
> cases, and are straightforward for even a relatively naive user to avoid.
>
> Thus, juju ensure-ha-state is almost equivalent to:
>
>     juju add-machine --jobs manager -n 2
>
> In my view, this command feels less "magic" than ensure-ha-state - the
> runtime implication (e.g. cost) of what's going on are easier for the
> user to understand and it requires no new entities in a user's model of
> the system.
>
> In addition to the new add-machine flag, we'd add a single new command,
> "juju machine-jobs", which would allow the user to change the jobs
> associated with an existing machine.  That could be a later addition -
> it's not necessary in the first cut.
>
> With these primitives, I *think* the responsibilities of the system and
> the model to the user become clearer.  Looking back to the original
> user questions:
>
> a) The "state manager" status of certain machines in the status is no
> longer something entirely divorced from user control - it means something
> in terms of the commands the user is provided with.
>
> b) The user already knows about destroy-machine.  They can manage broken
> state manager machines just as they would manage any other broken machine.
> Destroy-machine would refuse to destroy the any state server machine that
> would take the currently connected set of mongo peers below a majority.
>
> c) We already have add-machine.
>
> d) We already have destroy-machine. See c) above.
>
> REQUEST FOR COMMENTS
>
> If there is broad agreement on the above, then I propose that
> we start off by implementing ensure-ha-state with as little
> internal logic as possible - we don't necessarily need to put the
> transactional
> logic in to make sure it works without starting extra machines
> when many people are calling ensure-ha-state concurrently, for example.
>
> In fact, ensure-ha-state could probably be written as a thin layer
> on top of add-machine --jobs.
>
> Thoughts?
>
>

instead of adding more complexity and concepts, it would be ideal if we
could reuse the primitives we already have. ie juju environments have three
user exposed services, that users can add-unit / remove-unit etc.  they
have a juju prefix and therefore are omitted by default from status
listing. That's a much simpler story to document. how do i scale my state
server.. juju add-unit juju-db... my provisioner juju add-unit
juju-provisioner.

-k
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20131107/233b2983/attachment.html>