Linux Desktop Testing Project

[Matt Zimmerman]

On Fri, Jun 08, 2007 at 04:13:03PM +0300, Sivan Greenberg wrote:
> Matt Zimmerman wrote:
> > We can build (and initially run) this in a VM, until it's ready to deploy in
> > production, at which time we should be able to arrange for Canonical to host
> > it.
> 
> We probably need to provide accounts on that machine (the VM), to cater 
> for the scenario when I have for example written a test plan in dogtail, 
> used the bzr setup to put my testplan to work, but some setup in the OS 
> installed under the VM is not right or needs some modifications (such as 
>   additional package installation). I'd like to be able to fix that 
> setup, create a snapshot for further use and go on with my test plan.

I don't think that's appropriate; test development should happen locally,
and the production testing machine will be hands-off.  The test should be
completely self-contained, running in a clean environment.  Any dependencies
need to be handled by the tests themselves.  Most tests will start with a
completely fresh install.

> > example-content should be a good start; perhaps a good way forward would be
> > to add an example-content-extra binary package which would include lots more
> > for test purposes.
> 
> Hmm.. I'm thinking to scan through GNOME and other upstream products bug 
> trackers and collect content such specific images or media that caused 
> several known crash bugs (I even found one for eog and posted it) and 
> use this package to deliver those as a start.

It's good to have a variety of test files for many applications in any case.

> > I think it should be a manual step because false failures will be common
> > when testing desktop applications.
> 
> I wonder if we could identify those cases to be able to create an 
> "ignore list" to eventually allow testing cycles without intervention.

The problem isn't dealing with the failures; the test itself will be
relatively fragile (because it's tied to the UI), and so when that changes,
it will "fail" and need to be updated.  We should not ignore these failures,
but fix the tests.

> >> It would be great to have a setup where anyone could watch the tests 
> >> taking place as a webstream. Then anyone could see the latest crack of 
> >> the day and see how testing is going :)
> 
> We should probably talk to the Fedora testing projects folks, their 
> either had or still have something similar to this for the hive release 
> of Fedora.

I'd be interested to hear the details.

-- 
 - mdz