Running our autopkgtests in LXC/on armhf

[Martin Pitt]

Hello all,

I recently set up some Calxeda ARM nodes that we got for QA and
improved autopkgtest enough so that we can efficiently and robustly
run autopkgtests using the LXC runner [1]. As a first smoketest I run
all our 310 existing autopkgtests except linux and libreoffice on
them.

I wanted to see whether the four servers, LXC, and autopkgtest are
robust enough, and indeed they survived that. That's a major
improvement from our first attempt to run them on Panda boards, which
fell over more often than not.

I set that off yesterday evening, and by today noon the last
test has finished. Two of the nodes were done this morning already,
but two others had the bad luck to catch tests which hung
indefinitely and thus only timed out after 4 hours.

The full raw logs are at [2]. I went through all failures of packages
which don't fail in our production run with Qemu/KVM [3] and
categorized/summarized them, notes are at [4].

Summary
-------
  * 48 tests out of the 310 tests failed which don't fail with KVM on x86.

  * 34 failures happen on amd64 with LXC as well, with the same error.
    I. e. they don't fail because of ARM, but because the test only
    runs in a container and not in a full VM. (Note, we don't
    currently have a kvm-like qemu on ARM, so we have to use LXC there)

  * 14 failures are ARM specific, i. e. they succeed on x86/LXC.

  * 2 packages fail which also fail on x86/LXC, but they have
    additional errors.

So only 16 packages would need fixing to get their autopkgtests up to
par with x86. That's a lot less than I initially feared.

CI integration
--------------
We want to put this into production soon. To avoid blocking on LXC or
ARM/LXC specific failures, we will teach britney to only consider an
autopkgtest a failure if that test ever succeeded. In fact we should
do this for all architectures, as unfortunately 80% of the new
autopkgtests that we get fail (they hardly ever get tested in a proper
isolated environment by uploaders unfortunately).

Fixing tests
------------
In some cases tests that work in qemu fail in LXC because the test
tries to do kernel-level things like loading modules, accessing
cgroups, etc. These tests just can't run in LXC, and there is little
that we can do about it. We'll just continue running them in Qemu on
x86 and don't run them on ARM.

But in quite a lot of cases this is because the autopkgtest LXC runner
is stricter than the "null" runner that we are currently using; for
example, if your test has "Restrictions: build-needed" then in Qemu
your test still has all the build dependencies installed, but the LXC
runner resets the testbed between build and test; so your
debian/tests/control needs to specify all runtime Depends:. (Note that
you can use @builddeps@ in cases where this is appropriate)

Fixing tests to work in LXC is greatly appreciated, so if you want
your package to always work on ARM, and want dependencies that break
your package be held in -proposed instead of landing and breaking your
package, please make sure that you have a working LXC autopkgtest.

Locally running tests in LXC
----------------------------
Dependencies:

  $ sudo apt-get install lxc autopkgtest

Create an LXC container:

  $ sudo lxc-create -t ubuntu-cloud -n trusty-cloud  -- -r trusty

  (You can use any other name with -n, of course)

Run tests from package "pkgname" in the archive:

  $ sudo adt-run pkgname --- adt-virt-lxc --ephemeral trusty-cloud

Run locally fixed tests for package "pkgname" with the packages from
the archive:

  $ cd pkgname-1.1/
  $ # fix debian/tests/
  $ sudo adt-run -B --unbuilt-tree=. --- adt-virt-lxc --ephemeral trusty-cloud

  The difference between --built-tree and --unbuilt-tree only matters
  if your test has "Restrictions: build-needed"; then, use
  whichever state your tree is in.

Build locally fixed package, install its built debs for the test, and
run fixed tests from local source tree:

  $ cd pkgname-1.1/
  $ # fix whatever
  $ sudo adt-run --unbuilt-tree=. --- adt-virt-lxc --ephemeral trusty-cloud

Note that you can use the "-o /tmp/output-dir" option to adt-run if
you want to put the autopkgtest logs into a directory.

Let me know if you have any question.

Thanks, and happy weekend!

Martin

[1] https://plus.google.com/u/0/107564545827215425270/posts/HktYxb8XdMy
[2] http://people.canonical.com/~pitti/tmp/autopkgtest-arm/
[3] https://jenkins.qa.ubuntu.com/view/Trusty/view/AutoPkgTest/
[4] http://people.canonical.com/~pitti/tmp/autopkgtest-arm/analysis.txt

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20140124/79d72936/attachment.pgp>