[TEAM] Package Testing

[Pasi Lallinaho]

On 2014-11-14 15:15, Jack Fromm wrote:
> On 11/14/2014 06:22 AM, Elfy wrote:
>> Last cycle little testing was reported for packages, this cycle we 
>> want to try something different.
>>
>> While we have someone working on producing autopilot tests - but it's 
>> still early days with that.
>>
>> So exploratory testing.
>>
>> Plan to let people just use development release during the cycle, use 
>> application and report bugs to the tracker as they find them.
>>
>> In addition people can use the staging ppa for some applications in 
>> Trusty, Utopic and Vivid - so those new yet to be uploaded versions 
>> can be tested.
>>
>> Communication between the QA and Dev teams is important here - we'll 
>> call for testing on new apps when we've been asked to.
>>
>> We can run a few testing sprints during the cycle - nothing other 
>> than the though exists for this yet.
>>
>> Comments on this please.
>>
>> Thanks
>>

> Just using the development release has the advantages of being easier 
> for the testers and more realistic but it would also rely on having a 
> good cross-section of user types actually doing it.

While simply running the Xubuntu development version might be the 
simplest thing to do theoretically, many people who could be willing to 
report test results might not be able to do that; for example, because 
they only had one computer which is production-critical.

This is why we are offering the alternative of running updated software 
from PPA's, even on the non-development releases. If something breaks 
with a PPA package, it's relatively easy to disable the PPA and fall 
back to the version in the regular repository, especially comparing with 
having to do a fresh install.

> One disadvantage of that approach is that everything might not get 
> tested adequately.  I just counted and it looks like I would never 
> touch - or touch just once during the initial install - 20 of the 43 
> apps currently on the tracker.

That's the other side of exploratory testing. The other side is that the 
applications tester X uses get much more complex, real-life, daily 
testing than the applications would ever get if we always followed the 
testcases and only tested the features mentioned there.

> But if people are willing and able to run the new release during the 
> cycle (and report results), it would certainly be better than no 
> testing at all.

Again, this is why we are offering PPA's for other releases. And again, 
if that person with only one computer is willing to use the development 
PPA on their hardware and file any bugs they find during the whole 
release, it's (in my opinion) much better than them running every 
application test in a virtualized environment once.

Cheers,
Pasi

(Note: reversing top-posting by me)

-- 
Pasi Lallinaho (knome)                » http://open.knome.fi/
Leader of Shimmer Project             » http://shimmerproject.org/
Ubuntu member, Xubuntu team member    » http://xubuntu.org/