MRE request: lxc

[Martin Pitt]

Hey Stéphane,

Stéphane Graber [2014-04-02 16:19 -0400]:
> LXC upstream does full automated test runs on amd64, i386 and armhf for
> every commit to both the master and stable-1.0 branch. We also do test
> builds and test runs using clang, push daily builds to coverity for
> static code analysis and do automated cross-builds for Android.
> 
> The upstream Jenkins server is at: https://jenkins.linuxcontainers.org

I poked around and I can only find source builds for LXC itself, and
the daily download builds from lxc-templates. Where are the actual
test suite runs for CI?

I had a look at the test suite[1]. It covers quite a lot of the API,
but only the most common cases and often only ensures that the calls
don't fail: E. g. it doesn't check that a call like add_device_node()
actually does what it promises. But while it doesn't cover many corner
cases it certainly covers all the major workflows and functionality,
especially the integration tests like "lxc-test-ubuntu". These also
run in the autopkgtest which we intend to keep running for trusty
after the release.

I did a spot check of about 10 recent commits, and none of them were
accompanied by test updates (in particular tricky stuff like
https://github.com/lxc/lxc/commit/0d9acb9 which applies to relatively
subtle changes which can easily regress without noticing quickly).

Do you have plans to also cover testing of the templates? Bugs like
https://bugs.debian.org/743425 are a bit of a bummer to have if they
hit a stable release (apparently that's not what happened in that
case, though, it's just something which I ran into recently).

> Seeing the very large amount of SRUs we've been doing in the past
> few releases, it's my belief that an MRE will save a lot of time to
> Serge and I as well as the SRU team by simplifying the amount of
> documentation required for our updates.

I can't say that the existing test coverage would be sufficiently
reliable to use it as fully automated SRU testing, but for each new
microrelease we should run a test plan for the most important
scenarios. But with that I have no general objection, especially as
the changes you intend to do to the 1.0 branch are bug fix only
(contrary to other MREs).

So I don't see verifying individual bugs for SRUs as the most
important issue here, but regression testing. When doing that, and
limiting this MRE to bug fixes only, I'm fine with the MRE.

Thanks,

Martin

[1] Small chuckle: createtest.c creates a busybox container and says
"trusty container". Totally not important for this mail, of course :)

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/technical-board/attachments/20140403/8cb1b8ff/attachment.pgp>