Testing server packages with dep8 and multiple tests

[Robie Basak]

I think I have an issue running multiple tests. I want to add a dep8
test I've written to puppet. It already has a test, so I'm adding a
second one.

Initially my adt-virt-lxc driver didn't support the "revert" capability
and I found that I couldn't run both tests in a single adt-run.

adt-virt-null also appears to not support the "revert" capability. It
looks to me that this is what auto-package-testing uses, so I think if I
were to upload these tests then we'll end up with a test failure.

What I should do to fix the tests is to declare the restriction
"breaks-testbed". But then adt-run won't be able to use adt-virt-null to
run the test any more. It'll skip the test. That causes a non-zero exit
status, and from what I can see auto-package-testing will treat that as
a full failure.

I've since implemented "revert" and "revert-full-system" to the LXC
driver, so that gives us full support now. But of course the Jenkins QA
lab doesn't use that right now.

Have I got this right?

This seems important to me for the server use case, since just about
every test that tests a daemon will break the testbed. So I think we
need to be using a driver that supports "revert-full-system" in the QA
lab if we want to have more than one test per "daemon" package.

I think we should already be declaring "breaks-testbed" where the test
messes with a daemon's configuration, but if we do this now I think
we'll start failing everything.

Thoughts? Should I file a bug that says that multiple "breaks-testbed"
tests are not supported (whether they declare that or not)? And then
there are a slew of tests which should be declaring "breaks-testbed" but
do not, and we can't make them do so because that would cause us to
unnecessarily fail tests.