agent upgrading

[Kapil Thangavelu]

forgot to cc' the list ml, added.

On Mon, Jun 11, 2012 at 3:26 AM, roger peppe <roger.peppe at canonical.com>wrote:

> On 11 June 2012 01:11, Kapil Thangavelu <kapil.thangavelu at canonical.com>
> wrote:
> >
> > Looks good, a few questions on details embedded.
>
> Thanks for the useful response.
>
> > On Fri, Jun 8, 2012 at 1:07 PM, roger peppe <roger.peppe at canonical.com>
> > wrote:
> >>
> >> We'd like to be able to upgrade a running juju with new software.
> >> This is a scheme I originally came up with for upgrading
> >> minor (database-compatible) versions, somewhat modifed after
> >> discussion with Gustavo:
> >>
> >> Client:
> >>        - Push new version of tools.
> >>        - Set new global version number in state.
> >>
> >
> > what's the state path here of the version node?
>
> /version ?
>

sounds reasonable.. i was thinking /environment/version

>

>
> >> Machine agent:
> >>        - Wait for global version to change.
> >>        - Download new version.
> >>        - Copy version to where the local agents can see it
> >>
> >>        - Set new version in local agents' state.
> >>        - Point "current" symlink to new version
> >>        - Exit and let upstart start new version
> >
> >
> >
> > Not all agents on a machine maybe running on the same fs, a unit agent
> in a
> > container may need to download its own release.
>
> That's an interesting point. I had assumed that an outer container always
> has access to the fs of the contained items (LXC is essentially chroot
> + sugar, right?)
>
>
your right, the machine agent should have access. some things to keep in
mind though, it might be another volume though and its possible for a
container to be a different architecture (albeit the latter is a minor
concern).


> An alternative model would be to have all agents download the new
> tools independently
> (if that's the case, perhaps we shouldn't bundle them together, but have
> a separate URL for each agent/tool).
>
> > could you clarify what you mean by symlink?
>
> The idea is that there's a symlink for each tool which points to the
> current
> version of the tool. A tool upgrades itself by atomically redirecting that
> symlink to the new version of itself (that way we don't need to rewrite
> the upstart file each time the version is changed).
>
>
so a failure on the new release, would also entail reverting this symlink?


> > Which states are being recorded
> > in zk. ISTM, that we should have running versions, and proposed versions
>
> +1. I sketched over this bit, but that was part of the plan.
> Then you can see in the state when agents have successfully
> upgraded.
>
> > and
> > proposed timestamp recorded for all agents, so we can detect deltas from
> > outside the context of a given agent.
>
> I'm not sure what you mean by "proposed timestamp" here.
>

its hard to distinguish error from op in progress without explicitly
tracking the operation state or failing that a timestamp to differentiate
persistent errors.



>
> >> Other agent (provisioning or unit agent):
> >>        - Wait for agent's version to change.
> >>        - Point "current" symlink to new version
> >>        - Exit and let upstart start new version
> >>
> >> I think we should be able to do better than this. The problem is with
> >> the "exit and let upstart start new version" step - that means that if
> >> we happen to upload a broken version, then everything instantly breaks
> >> and needs manually restoring.
> >>
>
>> Here are some desirable features for an upgrade facility:
> >>
> >>        1. Uploading a broken tool shouldn't break anything.
> >>        2. ... even for a short while.
> >>        3. We shouldn't rely too heavily on upstart, given the possiblity
> >>        of ports to systems without upstart.
> >>
> >
> > re 3. its better to classify the functional property of such a system..
> it
> > sounds like your saying
> > that such other init systems might not have process management enough to
> > relaunch the agents if they die?
>
> No, I assume that we'll have something that can automatically start
> an executable on reboot, and if the agent crashes unexpectedly.
>


old school init.d systems don't nesc restart automatically, their originial
intention was boot. as a quick example.

$ sudo apt-get install redis-server
$ sudo kill redis-server
$ ps aux | grep redis-server

afaik, as far as auto restart, the common init like tools that support are
upstart, daemontools, systemd, and inittab.



>
> But I think it *might* be possible to use upstart events to do all
> the coordination between new and old versions - leveraging
> upstart in a deep way - and I think that would not be a good idea.
>

yeah, staying away from upstart implementation details for the high level
design sounds good.


>
> >> Obviously point 1 is not entirely attainable - a tool can be broken in
> >> any number of subtle ways that are not quickly detectable.
> >>
> >> However, if we each tool does a set of checks at startup time (checking
> >> the version in the zk database and other dependencies) then I think that
> >> the likelyhood of breakage can be drastically reduced.
> >>
> >
> > and it would then proceed with an upgrade if it detected a delta between
> > runtime and state version?
>
> The upgrade would not proceed if the version was incompatible.
>
> >> Here is a proposal for a scheme that addresses the above three points.
> It
> >> remains the same as the original scheme, with the exception instead of
> >> letting upstart start the agents directly, we interpose a intermediary,
> >> say "upgrader". This tool would be designed to be small, well verified
> >> and with minimal dependencies - designed to need upgrading very seldom.
> >>
> >>
> >> The final "exit and let upstart start new version" step is replaced with
> >> the following upgrade path.
> >>
> >>        - The agent asks the upgrader to run a new version of the agent
> >>        by passing it the name of an executable and arguments.
> >>
> >>        - The upgrader starts the new agent.
> >>
> >>        - The new agent connects to the state, does whatever verification
> >>        is necessary, and notifies the upgrader that it has successfully
> >>        started (but doesn't actually *do* anything yet).
> >>
> >>        - The upgrader notifies the original agent that the upgrade has
> >>        been successful.
> >>
> >>        - The original agent shuts down, notifies the upgrader that it
> >>        has done so, and exits.
> >>
> >>        - The upgrader notifies the new agent that it can continue,
> >>        picking up the work where the old agent stopped.
> >>
> >> A particular advantage of this scheme is that an upgraded agent
> >> will cause no down time, even if the new agent hangs for a long
> >> time when starting.
> >>
> >> If the agent exits without upgrading, then the upgrader tool
> >> will also exit, leaving upstart (or similar mechanism) to restart it;
> >> this provides a way to upgrade the upgrader itself.
> >>
> >> One possible drawback is that the new and the old
> >> agent running side-by-side might be a problem in
> >> resource-constrained environments. I don't think this would
> >> be a problem in practice (most resources will probably be taken
> >> as the agent continues to run, rather than at initialisation time),
> >> and we can work around it if necessary, by having a way to
> >> tell the upgrader to run the programs sequentially, and
> >> back off to the previous version if the upgraded version fails.
> >>
> >> I have a prototype of this proposal here (WIP, untested as yet):
> >>
> >>        https://codereview.appspot.com/6307061
> >>
> >> The upgrader in this implementation uses stdin and stdout to talk to
> >> its child processes;
> >> another mechanism could be substituted if desired.
> >>
> >> Major Version Upgrades (sketch)
> >>
> >> Major version (database) upgrades can fit into the above scheme by
> >> providing an additional synchronisation step. The client could give the
> >> global version a "pending" tag and wait for all agents to indicate that
> >> they are halted. Then it would upgrade the database, untag the version
> >> and let the upgrade proceed as usual.
> >>
> >
> > its not clear why you need a separate up-grader process, the two agent
> > processes can coordinate between each other so the new version can be
> > verified and the old one shutdown.
>
> That was my original plan, but I realised that AFAICS that's not compatible
> with the way that upstart (and presumably other supervision systems) work,
> as when the old process exits, upstart will think that it's terminated and
> start
> a new instance, even though a new instance has already started.
> It was not my plan to eliminate the use of upstart completely.
>
>

its unclear how using an upgrader process gets around that. upstart/init
sys still wants to track the new process pid. you could maybe go old style
init with a /var/run/agent-name.pid, but then again
having the new/old agents coordinating directly exactly replaces the
upgrader process afaics. your introducing the third party for the purpose
of coordination alone afaics.



> Another potential (though currently unrealised) advantage of a separate
> upgrader process is that it might allow us to automatically rewind to a
> previous
> version if a new version starts failing for some reason after succeeding
> initially. That may well be a crackful idea however.
>
>

the rewind should be possible till the new agent starts its activities,
even after its possible even without the upgrader, (ie. i've restarted 10
times in 5m, revert to old release), that's dicey though. say you do a
schema upgrade as part of the upgrade, reverting backwards is a hoser.


> > what happens when the new version of the agent fails to start? how is the
> > error handled/reported?
>
> The error would be logged. It's possible we might want to store the most
> recent error in the state, so that a juju info can quickly see what agents
> have
> failed to upgrade.
>
> > you'll need a barrier and some actor(s) to take on a coordinator role for
> > the db upgrade.
>
> I think my "pending" tagged version acts a barrier. I had indeed sketched
> over
> what component actually does the db upgrade, though I think that detail
> can be decided later - my aim in the "major version upgrade" section was
> merely to show in a hand-wavy kinda way that we can do major version
> upgrades without changing the basic upgrade mechanism.
>
>
sounds good, it should layer on top as additional coordination.

cheers,

Kapil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20120611/ac05001f/attachment-0001.html>