Feedback on the QA cycle
ub.untu at btinternet.com
Thu Mar 27 09:46:06 UTC 2014
On 20/03/14 23:38, Pasi Lallinaho wrote:
> this is a reply to the QA recap/feedback thread. As the original
> thread went off track, I decided to start a new one to discuss the
> original question at hand.
> PACKAGE TESTING
> First of all, I think it was a good move to run the package testing in
> groups and in cadence before we hit the beta milestones. Running all
> those tests and gathering a (big) list of bugs was and is important,
> especially now that we have entered the "bug fixes only" stage of the
> release preparing. I am sure we would be able to fix a lot less bugs
> that are annoying and affect numerous of people.
> That being said, I think the amount of calls was just about perfect
> for an LTS cycle. I personally think we should go through all the
> groups during regular releases as well, but possibly group more groups
> into one call, and relax on the amount of testing "required". Optional
> tests could be literally that; run if comfortable, but if they are
> left untested, that's fine as well.
> As to what (else) to test, I think we should try to focus on new
> features, as we did this cycle. This can and probably should be
> extended to running tests on applications that have had a major update
> during the cycle. All of this in a flexible manner; the more new
> things we have about to test, the looser running the other tests
> should be. Except on the LTS releases...
> I've yet to decide if some of the testcases are a bit too thorough or
> if they are just about right. I guess we can agree and assume that the
> amount of bugs is somewhat correlating with how deep the tests are. As
> I see it though, the deeper and specific the tests are, the more
> mechanic running them is. Which leads us to exploratory testing...
> I have a few doubtful thoughts on exploratory testing. How do we
> motivate people to run exploratory testing with the development
> version, while it is not ready for production, or day-to-day
> environments? If the tests aren't run on/as your main system, how can
> the testing be natural enough to be of exploratory nature? How do we
> specify a good balance between feature and exploratory testing?
> MILESTONE (ISO) TESTING
> It is hard to evaluate how the milestone ISO testing succeeded because
> we still have one beta to go, which is also the most important
> milestone. That is something where we can improve though.
> The alpha releases could have been focused more on specific issues.
> Now we kind of just ran through them without clear focus. Of course
> this means that developers need to have their stuff together earlier
> in the cycle, but that is a desirable direction generally.
> I would rethink the amount of alpha releases we want to participate in
> especially with non-LTS releases. We can opt-in for as many as we did
> now if we have set a clear point of focus for those. This looks
> unrealistic for T+1 though, as this cycle has been really busy for
> everybody and we have got a lot of stuff that was prepared in the last
> 2 years included.
> For the beta releases, we should get more publicity. We still have the
> beta 2 release to come, so let's try to fix at least some of that for
> To end the feedback on a positive note (though there weren't so many
> negative points in total anyway), I think we have been up to the
> highest possible standard with QA considering the size of our team and
> the amount of new things landing this cycle.
> Finally, a big THANK YOU Elfy for running the QA team, doing all the
> calls, reporting back to us, taking care of bugs being noticed,
> features landing in time et cetera... Last but not least, thanks for
> putting up with us all who have sometimes more or less neglected our
> duties in QA and being unresponsive to questions and calls. It is very
> much appreciated, and I totally think that 14.04 would be a lesser
> release without your work and persistence!
Rather than post to the last mail I'll reply to this one.
Thanks for the feedback by everyone - much appreciated :)
So I've taken this from the comments.
*Testcase grouping* - call for more than one at a time, I'll likely
be re-organising some of them post 14.04 as well.
**Optional testcases* - can leave these for non-LTS testing
*New feature testing* - much as we did this cycle, fit them in when
we can - existing testcases to take a back seat if new features need
*Exploratory testing* - I'm not looking at this any longer - or at
least, it needs to work in conjunction with autopilot testing, there
will be a mail to the list in the near future about this from one of
the other members of the QA team.
*Specific Testing during milestones* - Work specific package testing
into various milestones when it's appropriate for us. Necessarily
this will need to be led by devs - they'll know more about what
needs to be tested. Only take part in milestones when there is a need.
*Testcase feedback* - I'll send a mail to the list regarding this
seperately, those that have actually taken part in package testing -
your input on this will be invaluable, please join in with this
*Feedback* to the list does help us - but it is a whole lot easier
to follow the trackers, bugs reported to those end up on our
blueprints during cycles - we can track that. Mailing list threads -
not trackable. In addition when you are reporting to a tracker it
will tell you bugs that others have reported against that test, be
it a package or an image.
Ubuntu Forum Council Member
Xubuntu QA Lead
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the xubuntu-devel