Automatic testing of .deb's

[Ian Jackson]

(Note crosspost to debian-devel and ubuntu-devel.  If your reply does
not concern both Debian and Ubuntu, please remove the inappropriate
list.  And of course, in general, do not CC individual posters unless
they ask for it.)


Following discussions at Ubuntu's Montreal meeting, on debian-devel,
and elsewhere, I've done an initial implementation of my planned
automatic testing framework.  In many ways it is still a proof of
concept, and it lacks many useful features, but it can already be used
to test packages in a chroot in a basic way.

The package with the test machinery is `autodebtest'[1] and can be
found[2] in Ubuntu dapper and Debian sid.  The dependencies are
relatively conservative so I think the .deb should install and work
with breezy or sarge too.


There is currently only one test in one package.  I picked `mawk' as
an example because it is a package which is (i) very stable (ii) has
limited dependencies and (iii) has at least some upstream test suite.
The new test runs the provided upstream mawk tests (they are not
separated out by the upstream test suite so it looks like one test
from the autodebtest point of view).

This new version of the mawk package has not been published anywhere
yet.  So, I have appended the (small) patch to this message.

Next on my todo list for this project is to try to integrate piuparts
somehow.  This will mean inventing some machinery for `any package'
tests.


Should I file the mawk patch, attached, as a wishlist bug against
mawk[3] ?  It would be nice to be able to integrate the tests for each
package as far upstream as possible; at the moment, that probably
means putting it in the Debian packages.

I would like to have some idea what people think I should do with the
tests that we're hopefully going to have, eventually for lots of
packages.  Would Debian like those tests as patches in wishlist bug
reports, in general ?  That would seem to be best to me but before I
go down this route I'd like to be clear that that's what Debian
developers want.


Below you can find a demo transcript.


Thanks,
Ian.


[1] The package will be renamed `autopkgtest' in my next upload.
We apologise for the inconvenience.

[2] You need 0.5.2 which is currently awaiting archive processing for
both sid and dapper.  Currently autodebtest is in dapper universe; the
renamed package will be in main RSN.  You can find a copy here:
 

[3] I have BCC'd the Debian mawk maintainer.
 http://www.chiark.greenend.org.uk/~ian/autodebtest-tmp/


liberator:~> dchroot
Executing shell in 'dapper' chroot.
To execute a command as root, run "sudo <command>" and enter your password.

ian at liberator:~$ cd /work/AutomatedTesting/demo 
ian at liberator:/work/AutomatedTesting/demo$ mkdir unpatched
ian at liberator:/work/AutomatedTesting/demo$ cd unpatched/
ian at liberator:/work/AutomatedTesting/demo/unpatched$ dpkg-source -x /work/download/mawk_1.3.3-11.dsc
dpkg-source: extracting mawk in mawk-1.3.3
dpkg-source: unpacking mawk_1.3.3.orig.tar.gz
dpkg-source: applying /work/download/mawk_1.3.3-11.diff.gz
ian at liberator:/work/AutomatedTesting/demo/unpatched$ cd mawk-1.3.3/
ian at liberator:/work/AutomatedTesting/demo/unpatched/mawk-1.3.3$ debian/rules build >../build-log 2>&1
ian at liberator:/work/AutomatedTesting/demo/unpatched/mawk-1.3.3$ echo $?
0
ian at liberator:/work/AutomatedTesting/demo/unpatched/mawk-1.3.3$ fakeroot debian/rules binary >../binary-log 2>&1
ian at liberator:/work/AutomatedTesting/demo/unpatched/mawk-1.3.3$ echo $?
0
ian at liberator:/work/AutomatedTesting/demo/unpatched/mawk-1.3.3$ cd ../..
ian at liberator:/work/AutomatedTesting/demo$ mkdir patched
ian at liberator:/work/AutomatedTesting/demo$ cd patched/
ian at liberator:/work/AutomatedTesting/demo/patched$ dpkg-source -x /work/download/mawk_1.3.3-11.dsc
dpkg-source: extracting mawk in mawk-1.3.3
dpkg-source: unpacking mawk_1.3.3.orig.tar.gz
dpkg-source: applying /work/download/mawk_1.3.3-11.diff.gz
ian at liberator:/work/AutomatedTesting/demo/patched$ cd mawk-1.3.3/
ian at liberator:/work/AutomatedTesting/demo/patched/mawk-1.3.3$ patch -p1 -s </work/AutomatedTesting/mawk-1.3.3.adt.diff 
ian at liberator:/work/AutomatedTesting/demo/patched/mawk-1.3.3$ debian/rules build >../build-log 2>&1
ian at liberator:/work/AutomatedTesting/demo/patched/mawk-1.3.3$ echo $?
0
ian at liberator:/work/AutomatedTesting/demo/patched/mawk-1.3.3$ fakeroot debian/rules binary >../binary-log 2>&1
ian at liberator:/work/AutomatedTesting/demo/patched/mawk-1.3.3$ echo $?
0
ian at liberator:/work/AutomatedTesting/demo/patched/mawk-1.3.3$ logout
liberator:~> really dchroot
Executing shell in 'dapper' chroot.
root at liberator:~# cd /work/AutomatedTesting/demo/unpatched/
root at liberator:/work/AutomatedTesting/demo/unpatched# dpkg -i mawk_1.3.3-11_i386.deb
(Reading database ... 70650 files and directories currently installed.)
Preparing to replace mawk 1.3.3-11 (using mawk_1.3.3-11_i386.deb) ...
Unpacking replacement mawk ...
Setting up mawk (1.3.3-11) ...

root at liberator:/work/AutomatedTesting/demo/unpatched# logout
liberator:~> adt-run --output-dir ~/d --build-tree-tb /work/AutomatedTesting/demo/patched/mawk-1.3.3 --- adt-virt-chroot =dapper
mawktest             PASS
liberator:~> cat d/stderr-mawktest 
liberator:~> cat d/stdout-mawktest 
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan


testing input and field splitting
input and field splitting OK

testing regular expression matching
regular expression matching OK

testing checking for write errors
checking for write errors OK

testing arrays and flow of control
array test OK

testing function calls and general stress test
general stress test passed

tested mawk seems OK
liberator:~> really dchroot su -
(dapper) su - 
root at liberator:~# dpkg --purge --force-depends mawk
dpkg: mawk: dependency problems, but removing anyway as you request:
 ubuntu-minimal depends on mawk.
 base-files depends on awk; however:
  Package awk is not installed.
  Package mawk which provides awk is to be removed.
(Reading database ... 70649 files and directories currently installed.)
Removing mawk ...
root at liberator:~# logout
liberator:~> adt-run --output-dir ~/d --build-tree-tb /work/AutomatedTesting/demo/patched/mawk-1.3.3 --- adt-virt-chroot =dapper
mawktest             FAIL stderr: /tmp/tmp.9omUyv/mawktest: line 24: mawk:...
liberator:~> cat d/stderr-mawktest 
/tmp/tmp.9omUyv/mawktest: line 24: mawk: command not found
liberator:~> 


diff --exclude 'parse.[ch]' -ruN orig/mawk-1.3.3/debian/tests/control mawk-1.3.3/debian/tests/control
--- orig/mawk-1.3.3/debian/tests/control	1970-01-01 01:00:00.000000000 +0100
+++ mawk-1.3.3/debian/tests/control	2006-01-31 15:25:36.081499137 +0000
@@ -0,0 +1,2 @@
+Tests: mawktest
+Restrictions: rw-build-tree
diff --exclude 'parse.[ch]' -ruN orig/mawk-1.3.3/debian/tests/mawktest mawk-1.3.3/debian/tests/mawktest
--- orig/mawk-1.3.3/debian/tests/mawktest	1970-01-01 01:00:00.000000000 +0100
+++ mawk-1.3.3/debian/tests/mawktest	2006-01-31 15:25:36.081499137 +0000
@@ -0,0 +1,9 @@
+#!/bin/sh
+set -e
+sed -e '
+	/^PATH=/d;
+	/^mawk -W version/ s/$/ 2>&1/;
+' <test/mawktest >"$TMPDIR/mawktest"
+chmod +x "$TMPDIR/mawktest"
+cd test
+"$TMPDIR/mawktest"


--