Fwd: Recent improvements in autopkgtest, change of CI infrastructure next cycle

Thu Feb 27 16:56:01 UTC 2014

-------- Original Message --------
Subject: 	Recent improvements in autopkgtest, change of CI 
infrastructure next cycle
Date: 	Thu, 27 Feb 2014 17:49:19 +0100
From: 	Martin Pitt <martin.pitt at ubuntu.com>
Reply-To: 	ubuntu-devel-discuss at lists.ubuntu.com
To: 	Ubuntu Development Announcements 
<ubuntu-devel-announce at lists.ubuntu.com>

Hello Ubuntu developers,

(similar to the debian-devel-announce@ mail, but with some Ubuntu
specifics)

A lot has happened on the DEP-8 [1]/autopkgtest[2] front in the last months,
and today's autopkgtest 2.9 upload marks the completion of a lot of
long-standing feature, usability, and documentation development.

New documentation
-----------------
The manpages were overhauled to clarify what all the adt-run options do, and
some missing pages were added. There is now also a new document [3] which
introduces how to run autopkgtests in the most common use cases, as well as the
available virtualization servers (schroot, lxc, etc.) together with their
most important properties and pros/cons. This should be the first thing to read
for new users of autopkgtest, before diving into the individual manpages.

See /usr/share/doc/autopkgtest/README.running-tests.gz and the manpages for
adt-run and adt-virt-* for details.

This now also largely obsoletes the documentation at
http://packaging.ubuntu.com/html/auto-pkg-test.html.  In trusty+1 we
want to move away from the lp:auto-package-testing scripts towards
using adt-run with adt-virt-qemu, as this is simpler, can run
"breaks-testbed" tests, and tests packages more thoroughly by being
able to revert the testbed between builds and tests. We already use
the "new" approach on armhf and ppc64el, and I encourage developers to
use the new tools (and check above documentation) now.

New virtualization servers
--------------------------
autopkgtest now provides two new virt servers:

  * adt-virt-lxc: Run tests in containers. This comes with a script
    adt-build-lxc(1) which conveniently builds an appropriate container for a
    Debian or Ubuntu release. See the manpages for details, and
    README.running-tests.gz for an overview of this server.

  * adt-virt-qemu: Run tests in QEMU/KVM. This comes with a script
    adt-buildvm-ubuntu-cloud(1) which conveniently builds an appropriate VM
    image for an Ubuntu release. See the manpages for details, and
    README.running-tests.gz for an overview of this server.

    I'd really like to have a script to build a VM image for current
    Debian sid. As Debian doesn't have daily VM images to base this on, I
    appreciate ideas how to approach this.

See the respective manpages for details.

New test declarations
---------------------
Tests can declare one of the new "isolation-container" or "isolation-machine"
restrictions if running them in a schroot/container is not sufficient. Package
maintainers are advised to do that in their package test to avoid
confusing/alarming failures in CI if the test is known to need a stronger
isolation (for example, the udisks2 test can only work in a VM and thus will
get "isolation-machine").

Tests can now depend on "@builddeps@" which expands to the source package's
build dependencies. This is useful if you have many build dependencies which
are only necessary for running the test suite and you don't want to replicate
them in the test Depends:.

See README.package-tests.gz for details.

Usability
---------
adt-run's command line interface got some cleanup, dropping a few rarely used
and confusing options like --tmp-dir, providing short alternatives for common
options, improving the default behavior, allowing you to specify "schroot"
instead of "adt-virt-schroot", and more. Most scenarios can now be run with
only very few options/arguments. See README.running-tests.gz for details on that.

The --output-dir contents is much cleaner now.  Tests can now also specify
additional artifacts which get exported to the --output-dir, which is now
suitable for verbatim publication to CI frontends like debci
(http://ci.debian.net) or Jenkins.

adt-run now also uses dpkg-buildpackage to build packages instead of calling
the bare debian/rules build, which fixes long-standing problems with the
application of quilt patches and definition of dpkg-buildflags variables.

Self-Testing
------------
autopkgtest now has a set of integration tests which provide fairly good
coverage on various test scenarios and all runners except the Xen one (which
is currently an unmaintained step-child). This has greatly helped to identify
bugs, crashes, and missing corner cases, as well as allowing larger-scale
code rewriting.

http://ci.debian.net
--------------------
Debian now has a service similar to our
https://jenkins.qa.ubuntu.com/view/Trusty/view/AutoPkgTest/:
http://ci.debian.net continuously runs package tests declared in the
DEP-8 format, using autopkgtest with the schroot runner (soon to be
moved to LXC). This actually reproduced a lot of the failures that
we got in armhf/ppc64el as these run tests in a rather minimal
debootstrap container instead of the relatively fat ubuntu-cloud
images with lots more packages pre-installed. In the medium term this
will hopefully lead to fewer broken autopkgtests getting imported, and
we can more easily work with Debian on fixing the tests or adding new
ones.

Contact
-------
As usual, if you find a bug or have a suggestion for a small improvement,
please file a bug against autopkgtest. If you want to help out with autopkgtest
development or the services around it (like http://ci.debian.net), or want to
discuss bigger new features, please do that on the mailing list [4].

Thanks for your attention!

Martin Pitt

[1] http://dep.debian.net/deps/dep8/
[2] http://metadata.ftp-master.debian.org/changelogs/main/a/autopkgtest/unstable_changelog
[3] http://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=blob_plain;f=doc/README.running-tests
[4] http://lists.alioth.debian.org/mailman/listinfo/autopkgtest-devel

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-quality/attachments/20140227/3cdf327d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 820 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-quality/attachments/20140227/3cdf327d/attachment.pgp>
-------------- next part --------------
-- 
ubuntu-devel-announce mailing list
ubuntu-devel-announce at lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-announce