Feedback on the QA cycle

[Pasi Lallinaho]

Hello,

this is a reply to the QA recap/feedback thread. As the original thread 
went off track, I decided to start a new one to discuss the original 
question at hand.

PACKAGE TESTING

First of all, I think it was a good move to run the package testing in 
groups and in cadence before we hit the beta milestones. Running all 
those tests and gathering a (big) list of bugs was and is important, 
especially now that we have entered the "bug fixes only" stage of the 
release preparing. I am sure we would be able to fix a lot less bugs 
that are annoying and affect numerous of people.

That being said, I think the amount of calls was just about perfect for 
an LTS cycle. I personally think we should go through all the groups 
during regular releases as well, but possibly group more groups into one 
call, and relax on the amount of testing "required". Optional tests 
could be literally that; run if comfortable, but if they are left 
untested, that's fine as well.

As to what (else) to test, I think we should try to focus on new 
features, as we did this cycle. This can and probably should be extended 
to running tests on applications that have had a major update during the 
cycle. All of this in a flexible manner; the more new things we have 
about to test, the looser running the other tests should be. Except on 
the LTS releases...

I've yet to decide if some of the testcases are a bit too thorough or if 
they are just about right. I guess we can agree and assume that the 
amount of bugs is somewhat correlating with how deep the tests are. As I 
see it though, the deeper and specific the tests are, the more mechanic 
running them is. Which leads us to exploratory testing...

I have a few doubtful thoughts on exploratory testing. How do we 
motivate people to run exploratory testing with the development version, 
while it is not ready for production, or day-to-day environments? If the 
tests aren't run on/as your main system, how can the testing be natural 
enough to be of exploratory nature? How do we specify a good balance 
between feature and exploratory testing?

MILESTONE (ISO) TESTING

It is hard to evaluate how the milestone ISO testing succeeded because 
we still have one beta to go, which is also the most important 
milestone. That is something where we can improve though.

The alpha releases could have been focused more on specific issues. Now 
we kind of just ran through them without clear focus. Of course this 
means that developers need to have their stuff together earlier in the 
cycle, but that is a desirable direction generally.

I would rethink the amount of alpha releases we want to participate in 
especially with non-LTS releases. We can opt-in for as many as we did 
now if we have set a clear point of focus for those. This looks 
unrealistic for T+1 though, as this cycle has been really busy for 
everybody and we have got a lot of stuff that was prepared in the last 2 
years included.

For the beta releases, we should get more publicity. We still have the 
beta 2 release to come, so let's try to fix at least some of that for 
Trusty.

CONCLUSION

To end the feedback on a positive note (though there weren't so many 
negative points in total anyway), I think we have been up to the highest 
possible standard with QA considering the size of our team and the 
amount of new things landing this cycle.

Finally, a big THANK YOU Elfy for running the QA team, doing all the 
calls, reporting back to us, taking care of bugs being noticed, features 
landing in time et cetera... Last but not least, thanks for putting up 
with us all who have sometimes more or less neglected our duties in QA 
and being unresponsive to questions and calls. It is very much 
appreciated, and I totally think that 14.04 would be a lesser release 
without your work and persistence!

Cheers,
Pasi

-- 
Pasi Lallinaho (knome)                      » http://open.knome.fi/
Leader of Shimmer Project and Xubuntu       » http://shimmerproject.org/
Graphic artist, webdesigner, Ubuntu member  » http://xubuntu.org/