Fwd: Re: About dep-8-tests

[Jamie Strandboge]

Hi, sorry for the delay. The team discussed this a bit and I'm
forwarding Marc's response to the thread, which I agree with.

-------- Original Message --------
Subject: Re: About dep-8-tests
Date: Wed, 05 Jun 2013 08:09:12 -0400
From: Marc Deslauriers <marc.deslauriers at canonical.com>
To: security at ubuntu.com

On 13-06-04 06:16 PM, Seth Arnold wrote:
> I figured we should respond to the server team soonish about this, but
> wanted to think about it a bit myself before contributing.. amazing how
> time flies.
> 
> Anyway, some thoughts amongst us ..
> 
> On Tue, May 21, 2013 at 04:26:59PM +0200, Yolanda Robla wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi
>> In server team we have been working in adding dep-8-tests in several
>> packages:
>> https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-seeded-qa-workflow
>>
>> That will improve QA because every time a new package version is
>> pushed, or some dependency changes, tests will be run and will make
>> sure that packages are still working correctly.
>>
>> But we are finding some conflicts in that process. For the following
>> packages:puppet, open-iscsi, solr-tomcat, exim4, mailman, squid,
>> vsftpd, backuppc, lxc, snmp, freeradius...
>>
>> it will be useful to reuse the tests that are already in QRT branches:
>> https://code.launchpad.net/~ubuntu-bugcontrol/qa-regression-testing/master
>>
>> As they are providing the same functionality that we need. This leads
>> to the problem of code duplication, because the tests will be
>> duplicated in QRT branch and package branch, that will make that more
>> difficult to maintain.
> 
> Our tests have the potential to be destructive. On the one hand, I hope
> our testers are able to handle unsafe code, but I don't believe they were
> intended to handle active adversarial content, even if we do expect the
> tests to fail. They might not.
> 
> A lot of our tests actually start up multiple services on ports, sometimes
> with configurations that we wouldn't want exposed publicly. Granted the
> testers are probably intended to be firewalled, but this would still be
> a concern to me.

Yes, I do believe they are run in a test VM on jenkins though, the
lp:auto-package-testing repo contains the scripts that start a new vm,
or a new lxc instance.

> 
>> We are also neededing to copy testlib.py file
>> along all the tests. Packaging testlib.py and using that as a depends
>> should be the cleanest option, but this code is not thought to be
>> production code and is not 100% guaranteed. Should we package it or
>> find another alternatives? - also these tests need to go to Debian, so
>> in the solution we find we need to consider it -
> 
> I'd be leery of packaging this in anything other than a PPA. It just
> isn't intended to be publicly consumable code.

I think including it in the package itself by duplicating is the best
approach, since we're not guaranteeing any api stability in that...

> 
>> Also there is the problem with UTAH tests. If security team is working
>> on some UTAH tests we should discuss the way to integrate them with
>> dep-8-tests.
>>
>> Finally there is another issue with older releases. If some package in
>> saucy has dep-8-tests, and the precise version doesn't have it, how
>> should we act? How can these tests be run, maybe we should backport them?
> 
> This is my biggest concern. Our tests are relatively easy to extend once
> and benefit from for all five or six releases at once. If something
> needs to be special-cased for future or past, it's not -too- hard. But
> there's only one place to maintain it.
> 
> Having to add identical tests to five versions of a package just to add
> one more test would be moderately annoying.

We are definitely not going to be maintaining the autopkgtest versions
of these tests. Whoever commits to doing that is going to be on the hook
for updating them if they wish.

> 
>> So i think we should have a discussion about these topics and find the
>> best way to deal with them.
> 
> My inclination says that individual test cases should be cherry-picked
> from QRT based on the merits of the individual test. We get enough test
> coverage in the packages to provide a smoke-test for general use, but
> keep our QRT tests separate, so we can more easily continue to add to
> them.
> 
> Of course, if there are changes made to extend the tests to doing useful
> things in the dep-8 test versions, it might be nice to bring those back
> into QRT too -- or at least make it easy for us to run autopkg tests as
> part of our routine. (Maybe we should _move_ the QRT tests that get
> cherry picked, just to avoid excessive runtimes if we use both QRT and
> autopkgtest at update publication time?)
> 

Yes, we should probably see how we can run the autopkgtests when doing
our QA. As for using our QRT tests, in most likelyhood they will be
using a subset of our scripts, as we have some interactive stuff, and
some destructive stuff that they will probably be commenting out.

I feel we should continue doing as we do, and ignore the fact that
someone is forking our tests. Maintaining our tests in packages
themselves for stable releases is madness.

Marc.







-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 899 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-quality/attachments/20130610/d7dda5db/attachment.pgp>