API compatibility policy and practices between juju versions

[Curtis Hovey-Canonical]

I am not sure if I am leading a discussion or just stating that we
have a problem that I don't believe can be ever solved.

We abandoned the release of 1.16.4 because we found it was
incompatible with 1.16.3
    https://bugs.launchpad.net/juju-core/+bug/1252469
    "API incompatability: ERROR no such request "DestroyMachines" on Client"

I now believe this bug is in the same class of problem:
    https://bugs.launchpad.net/juju-core/+bug/1250154
    "ERROR no such request "EnvironmentGet" on Client"

I suspect both bugs also mean that 1.18.x will be incompatible with 1.16.x

1. I think Juju selects the latest version at the patch-level because
they are always cli/api compatible between clients and servers. We
*require* 100% api/cli compatibility. This is the first bug.

2. I believe enterprises like our own IS will stagger minor upgrades.
Over a period of weeks, 1.18.0 will be used with 1.16.3 and they
require api/cli compatibility. This is bug two.

Enforcing point 1 is tremendously hard. We would have an ever growing
matrix of tests for the juju client-server combinations + a test for
every patch landed to verify that the user can accomplish the task. We
don't have the staff to do this. We are taking risks with each patch
release.

We are not enforcing point 2 at all. CI has found some bugs because it
nominally acts like devops when it runs upgrade tests. Even when the
upgrade fails, we attempt to scp the logs from the failing machine
just like a devop would.

Speaking for CI, we would never want a feature removed in a single
release. The feature needs to be deprecated so that we. like all IS
departments, have time adjust scripts as we transition from one
version to the next.

I think deprecated-for-one-release is hard. in the case of API every
where, I think that means that we add api, but fallback to cli so that
things like scp and terminate-machine always work across versions. The
old way of doing something is removed after one minor release.

I personally think a non-matching tools on bootstrap/depoy is crack.
At least warn that the client and server do not match. I think we can
promise status and upgrade-juju works between versions, allowing the
devops to do the upgrade. The aborted 1.16.4 can do that, so I helped
others build their own package and setup the private cloud to use just
the version so that they could switch everything to the version that
works best for them.



-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui