Fwd: Re: Juju Stable Update Policy

[Martin Pitt]

I just realized that Curtis didn't reply to the list. Forwarding.

Martin
----- Forwarded message from Curtis Hovey-Canonical <curtis at canonical.com> -----

Date: Mon, 28 Jul 2014 12:47:11 -0400
From: Curtis Hovey-Canonical <curtis at canonical.com>
To: Alexis Bruemmer <alexis.bruemmer at canonical.com>, martin.pitt at ubuntu.com
Subject: Re: Juju Stable Update Policy
X-Spam-Status: No, score=-1.9 required=3.4 tests=BAYES_00 autolearn=no version=3.3.2

>> From: Martin Pitt <martin.pitt at ubuntu.com>
>> Date: Tue, Jul 22, 2014 at 9:05 AM
>> Subject: Re: Juju Stable Update Policy

>> Alexis Bruemmer [2014-07-08 14:05 -0700]:
>>> > > Juju developers and have a specific QA review, and a variety of fresh
>>> > > install and upgrade combinations must be explicitly tested on every
>>> > > cloud
>>> > > that Juju supports (for details see the “Testing a revision” section
>>> > > of
>>> > the
>>> > > Juju CI Design and Operation document linked below)
>>> >
>>> > This really seems to be the crux of regression testing as far as SRUs
>>> > and existing cloud deployments are concerned. Whenever I try to open
>>> > the google docs I just get a "File unavailable. Sorry, there's a
>>> > problem with this file.  Please reload.", so I can't look at the
>>> > details. Does that mean that newer clients are tested on existing
>>> > deployments with older servers and agents? Which combinations does
>>> > that cover?
>>> >
>>>
>>> I have updated the sharing policies for the Juju Ci doc, let me know if
>>> you
>>> still have issues accessing the file.
>>
>> I can see it now, but it still doesn't explain how you test backwards
>> compatibility to existing deployments? Or I can't find it.
>>
>> I can see teh google doc now, but it still crashes on the first mouse
>> click. I think for long-term documentation we don't want to bury that
>> in google docs, but perhaps underneath
>> https://wiki.ubuntu.com/StableReleaseUpdates ?
>>
>> So in summary, I have reasonably good confidence in the currently
>> documented CI, but I need some more convincing about backwards
>> compatibility testing.

We don't have automated tests of minor-to-minor-client-to-server yet.
We are not scheduled to start work on this feature for a a few weeks.

We are added infrastructure to Juju CI to support this and we have
done some manual testing to prove how we will automate this.

1. CI will have a store of stable minor jujus: 1.18.4, 1.20.1, 1.21.0...
2. For each new revision of juju,
    * For each stable minor juju an env will be bootstrapped.
      * The juju client under test will be run through a series of
        commands that demonstrate the env can be maintained.
      * The test will culminate in a destroy environment
         to show the env can reach its EOL.
    * An env will be bootstrapped with the new juju
      * For each stable minor juju
        a series of commands will test that old client can
        maintain the new env.
    * reports.vapour.ws will have a report showing the all combinations
      pass test suite.
3. Only revisions that pass all tests are considered for release,
    CI will provide the actual release tarball.
4. CI will gain a mechanism to test a packages built for Ubuntu
    (ubuntu's packaging rules) that will test the packages pass all the
    tests the release tarball passed. Demonstrating that juju and the ubuntu
    packaging are compatible with the versions published in Ubuntu.


-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui

-- 
technical-board mailing list
technical-board at lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/technical-board

----- End forwarded message -----

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)