Bootstrap scheme for Go port
kapil.thangavelu at canonical.com
Tue Apr 24 14:13:24 UTC 2012
Excerpts from Clint Byrum's message of 2012-04-23 09:40:41 -0700:
> Excerpts from Kapil Thangavelu's message of Mon Apr 23 08:01:12 -0700 2012:
> > Excerpts from Gustavo Niemeyer's message of 2012-04-23 05:51:32 -0700:
> > > On Mon, Apr 23, 2012 at 03:53, Kapil Thangavelu
> > > <kapil.thangavelu at canonical.com> wrote:
> > > > Looks great, we need this. Two comments though, since the version is
> > > > effectively the same through the cluster (minus errors) what do we gain by not
> > > > just using a single integer ala charms, ie. even minor version code drift is
> > > > potentially problematic.
> > >
> > sem-versioning here requires human interpretation and is error prone for
> > compatibility evaluation.
> s/requires/requires and enables/
> Its a feature, not a bug. :) Rather than expecting we can get 100%
> test coverage for this, we should make a best effort, but then enable
> a release manager to step in and make things right.
> > We already have a few examples from the last cycle of either backwards
> > compatible state changes accompanied by incompatible code changes or
> > incompatible state changes, in both cases the developers thought they where
> > compatible.
> And as juju now moves from "rapid early evolution" to "widespread use"
> those changes would be *reverted* in a stable release.
via roll forward to a new stable revision.
> > i'm suggesting rather than work via a 4 part version string and possibly a
> > testing infrastructure to validate the correctness, we just utilize a monotonic
> > increasing int. I'm also assuming the implementation of schema upgrades though
> > as part of this feature, such that the distinction between revs is varied based
> > only on the presence of an associated migration to the revision.
> This would chase away any actual users pretty quickly. Users expect that
> software will break every once in a while in a way the authors never
> dreamed of. As a user of software though, I expect that these reports
> will be responded to quickly, and fixes will be considered.
agreed any non trivial software always has bugs. so i think your
saying semver allows users and a release manager to employ risk management
with flexibility wrt to version increments across major/minor/revs by defining
compatibility points. as for users running away, i can think of a few other
software pieces that are seeing some adoption that use int versioning
(sublimetext, systemd), granted their exceptions to the norms. doing rel-eng on
a single release in response to critical bugs is much faster then doing them
across many release branches.
> If we go with monotonically increasing numbers, we have no way to keep
> moving forward *and* respond in a clear manner:
> * 2.2.0 releases with new features over 2.0.0
> * 2.3.0 opens, massive new feature lands with huge refactoring in code
> * security problem, 2.0.2 and 2.2.2 released with patch.
> * 2 weeks pass, users download 1.2.0 and install it for testing,
> critical backward incompatible break is found.
> * 2.2.4 released with backward incompatible break fixed
> * 2.3.0 branch already has backward incompatible behavior as author
> intended, now adds documentation of the difference + migration
> code and/or advice.
> * 2.3.0 becomes 3.0.0 because of backward incompatible break.
This also goes along with managing multiple release branches and multiple paths
for env evolution to a current version, ie a veritable graph of transitions of
state upgrades to verify, which works given enough resources.
> With monotonically incrementing integers...
> * 6 "releases" with new features
> * 7 introduce massive new features/refactoring
> * security problem, 5.1, 6.1 released with patches
version 8 introduced with security fixes, the int always roll forward.
> * 2 weeks pass, users download 6 or 6.1, testing, critical, etc.
> * release 6.2 with backward incompatible break fixed.
> * ... much the same, ending with a '7' releasing w/ the features and
> backward incompatibility.
backwards compatibility is critical in the context not of the internal
juju code, but of the public api exposed and the compatibility questions posed
by cli or charms. in this respect semver has some value, as it
talks about the gradual introduction of deprecation before removal of
compatibility. but we always want to preserve environments, so when is breaking
backwards compatibility an option?
but let's review the consumption of state & apis ..
client -> juju environ code -> charm
client to juju currently has a circuit breaker in the form of topology
versioning, but given the direct state manipulation by the client this is
fairly dangerous for any version drift between client and the environment code,
ie introduction of subtle inconsistencies. the charm to juju interface is also
unversioned. so this spec which is focused on versioning the env code,
addresses neither of the backwards compatibility issues wrt to api consumers
indeed as juju matures we need to start assuming multiple clients of
differing versions. afaics the best way to do that is to start disassociating
the client from state management to a rest api that's is url versioned to a
backend impl, and to start tagging charms with juju cli api version. ideally
separating the client and env server/agent code into separate packages,
effectively this proposal is already saying the same as the env code eschews
distro package versions.
The same applies to charms already extant in the environment as services. their
written against particular versions of the juju cli api, which aren't captured
either by juju or the charms. To date we've always been additive wrt to changes
to the cli api, but that doesn't help deploying a newer charm against an older env.
digressing note, the versioning here isn't as clean as the client api versioning
as we'll need to multiplex cli tools and hook contexts (symlink to temp add to path
works well, multiple charms in a container can be executing against different
charm api versions).
in charms we encourage fact based discovery when executing upgrades hook (see
bug: http://pad.lv/766721 ) hooks, we could employ the same here but at a more
explicit level via charms defining their api version, and juju defining which
api its supports .
ie. backwards compatibility should always be preserved by carrying forward api
versions, and then deprecating on a time basis, analogous to how we support
distro versions for known set times, except in this case its not code support
but api support.
so in addition to the version moniker to tarball mapping the release url
endpoint could also hold supported api versions against each release (client &
charm). deprecation takes place at known points in time *and* we can verify the
environment for suitability of an upgrade when an api support is dropped (at
least for charms).
getting back to the juju environ code, what defines breaking backwards
compatibility here for an upgrade.. the same code version is extant
throughout the environment, so its not with itself. the primary issue then is
state upgrades, which is why i think they should be defined from the get go for
an upgrade procedure. juju isn't a framework ala a cms etc, so we don't have
user extensions modifying the codebase that need more indepth versioning
support and which require manual upgrade modifications. Its also not a critical
service from the perspective of user/application usage of the environment, ie.
the underlying services in an environment remain up during an upgrade.
> Now, as a user, how do I know that 6 wasn't supposed to introduce
> backward incompatible changes with 5 and that it is safe to upgrade to
> it to take advantage of the new features?
> How do I know that I need to run migrations from 6 -> 7?
migrations are automatically run as part of an upgrade.
> How do I automate security updates?
By always making it safe to upgrade, enabling users to be agile about upgrading,
ala continous deployment.
> The scheme set down with all 3 parts is able to satisfy all of those
> needs pretty easily I think.
It introduces complexity and extra work in the form of release branch
management and multiple state upgrade paths to support. Its socially common
usage i know for software, but even with the monotonic int approach the end goal
should be the same ie. defined backward compatability for critical usage as
a saas/paas/iaas provider. if that's a goal then while semver feels more
flexible its not clear that its not just adding aditional work. ie. the
charm api and client api versioning is already needed regardless of the core
version scheme. metadata based captures of api compatibility in a juju releases
give more precise information wrt compatibility for a given charm or client
then a major/minor/patch increment imo. the alignment of time based
api support compatibility to distro support cycles is just a cadence bonus.
The flip-side to that is establishing the importance of api versioning in the
go-port from the start.
let me put it another way, if semver is needed for juju why isn't it needed for
charms? another digression, we could say that upgrading charms is also broken
as it doesn't take into account the interface changes for a service.
usage stable/unstable/dev come out to being various release channels (ie.
resource urls), i'd be happy to expound on that if its useful.
feel free to tell me i'm on crack ;-) it was a fun gedanken experiment, and
hopefully useful regardless of the versioning scheme.
More information about the Juju