Linux Desktop Testing Project

[Ian Jackson]

Henrik Nilsen Omma writes ("Re: Linux Desktop Testing Project"):
> I'm very interested to get a pilot testing project up and running during 
> the gutsy cycle. We should get a machine set up somewhere that would 
> just continuously run tests. It would run through a list of test 
> scripts, then run the update manager and then start from the top.

I did have a setup running at home doing package testing until a
complicated hardware logistics meant that the machine's motherboard is
now not supported by any Ubuntu Xen kernel.  We're going to be moving
this to the datacentre just as soon as I can test the glue I'm giving
to the sysadmins.

> I think the main challenges at this point are:
> 
>  * Writing sensible test scripts for all the applications we care about. 
> Some exist, but are these realistic enough to do sensible testing?

This is the biggest job.

>  * Where to run it? Presumably an efficient test setup needs one or more 
> dedicated machines.  We would commit test scripts to a bzr repository 
> that the machine would regularly pull from.

My autopkgtest machinery seems like the right answer to this.  We'll
have a machine in the DC which contiuously seeks out a package to test
and installs it and runs its declared tests.

We'll have to provide a standard answer to `this is a GUI application
which uses dogtail, please install dogtail too and run the test in a
desktop application context' but that shouldn't be too troublesome.

For this, it would be really nice if tests were done so that they
provide the autopkgtest package test interface.

>  * Filing bugs - when a test fails to produce the right output, how is 
> hat information brought into our normal bug workflow? Does it start on a 
> webpage and/or email notifications from which we manually write bugs or 
> do we want a more automated procedure (there is a danger of getting too 
> many false bugs). What's the workload of doing it manually?

My plan for autopkgtest is to have it initially report to me by email
and I'll see what to do with the reports - probably, put them on a web
page.

> It would be great to have a setup where anyone could watch the tests 
> taking place as a webstream. Then anyone could see the latest crack of 
> the day and see how testing is going :)

Indeed.

Ian.