Integration of autopkgtest with proposed-migration

[Colin Watson]

Some of you have already noticed that the process which migrates uploads
from the proposed pocket to the release pocket once they're ready [1]
[2] is now also kicking off runs of autopkgtest [3].  I'd been holding
off announcing this for a while so that we could get things working a
little better, but it's really about time to explain what we're doing.

Once an upload is sufficiently ready in -proposed (that is, it has no
out-of-date binaries on amd64 and i386), proposed-migration will ask a
Jenkins instance to run autopkgtests for that package and for its
reverse-dependencies.  This allows package maintainers to check not only
that their package builds and passes tests in its build tree, but that
it works properly as installed from .debs, and that it continues to work
properly when the packages it depends on change.  Uploads will not be
allowed into the release pocket until all relevant tests have passed or
until any failures have been manually overridden by a member of the
release team.

Developers will be sent mail about autopkgtest failures in their
uploads.  You may have received some of these already.

Some frequently asked questions:

 * Why are you doing this?  It makes it harder for me to land my
   changes.

   The whole proposed-migration process is part of a balance between
   quality (only land changes that work) and velocity (make developers
   more productive by letting them get changes to users more quickly).
   Some of it, including this work, is loosely inspired by suggestions
   from Scott James Remnant [4].  I think there's widespread agreement
   that the introduction of proposed-migration has helped to make
   Ubuntu's development release very much more stable and useful.  There
   are several classes of problems that it could never have caught in
   its previous form, though, and by integrating autopkgtest I hope that
   we can address that without slowing things down too much.

   In the best case, an upload with associated autopkgtests will take an
   extra publisher cycle to reach the release pocket.  It may take
   longer if its tests take longer to run.  I think this is generally
   acceptable, but I also think velocity is very important [5], and I'm
   watching the process closely at the moment to try to minimise the
   extent to which it holds anyone up.

 * Why not only require tests to pass if they've passed before?

   The main part of proposed-migration works on a ratchet: the result of
   promoting a package must leave the number of uninstallable packages
   in the archive no worse than before.  For this work, though, I chose
   to take a more absolutist approach: tests must pass even if they
   previously failed.  This is mainly because there are few enough
   autopkgtests that I don't think it's an unreasonable request to fix
   them; at the time of writing we have 22 failing jobs out of 166 total
   [6].

 * But I really need to land this change now!

   Members of the ubuntu-release team can override either a failing test
   for all the uploads that caused it to be run ("force-badtest") or a
   single upload with some failing tests ("force-skiptest").  We hang
   out on the #ubuntu-release channel on freenode, so come and explain
   to us why it's more urgent to land the change than to fix the
   failure.

 * How do I tell what's going on with a test?

   The "excuses" page has links to the Jenkins logs for each running
   test.  Unfortunately jobs run on a private Jenkins instance and are
   only copied over to the public one after they complete, so if you
   want to find out what's going on with a job in progress then you need
   to ask a Canonical employee with QA lab VPN access [7] to look at the
   private instance for you.

 * Why do you run tests for reverse-dependencies?  Won't that take ages
   for core packages?

   Running tests for a package itself is somewhat useful, but doesn't
   gain all that much; most such tests could in principle be run as
   build-time tests, although there's some benefit in running them in a
   different environment, based on the installed version of the package
   rather than what's in the build tree, and with only their run-time
   dependencies.  The real benefit comes from cross-package testing.
   This means that you get to put assertions in your package that your
   dependencies work, and have that tested when those dependencies
   change in such a way that they only get to migrate to the release
   pocket once the whole assembly is sound.

   That said, we only currently run tests for direct
   reverse-dependencies, not indirect ones (e.g. ubiquity -> python3 ->
   python3.3), so there are some gaps in our coverage here.  We may
   change the details of this behaviour over time.

   It is possible that there are indeed packages for which this will
   take ages.  If this becomes an unreasonable problem, we can force the
   packages in question.  At the moment, we seem to have plenty of
   capacity for running tests.

 * How do I reproduce test failures?

   You can use the "adt-run" command in the autopkgtest package, which
   supports various back-ends (for instance, I usually use "adt-run
   foo.dsc --- adt-virt-schroot saucy-i386"), or you can use the more
   comprehensive tools in lp:auto-package-testing.  For details, see the
   Ubuntu Packaging Guide [8].

Many thanks to Jean-Baptiste Lallement for doing the lion's share of the
hard work for this.

[1] https://lists.ubuntu.com/archives/ubuntu-devel/2012-October/036043.html
[2] http://people.canonical.com/~ubuntu-archive/proposed-migration/
[3] http://dep.debian.net/deps/dep8/
[4] http://netsplit.com/2011/09/08/new-ubuntu-release-process/
[5] http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/ubuntu/2011-10-24-quality-in-12-04.html
[6] https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/
[7] https://wiki.canonical.com/UbuntuEngineering/QA/VPN
[8] http://developer.ubuntu.com/packaging/html/auto-pkg-test.html

-- 
Colin Watson                                       [cjwatson at ubuntu.com]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 892 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-devel-announce/attachments/20130704/03973ce1/attachment.pgp>