Feedback on the QA cycle

[Elfy]

On 20/03/14 23:38, Pasi Lallinaho wrote:
> Hello,
>
> this is a reply to the QA recap/feedback thread. As the original 
> thread went off track, I decided to start a new one to discuss the 
> original question at hand.
>
> PACKAGE TESTING
>
> First of all, I think it was a good move to run the package testing in 
> groups and in cadence before we hit the beta milestones. Running all 
> those tests and gathering a (big) list of bugs was and is important, 
> especially now that we have entered the "bug fixes only" stage of the 
> release preparing. I am sure we would be able to fix a lot less bugs 
> that are annoying and affect numerous of people.
>
> That being said, I think the amount of calls was just about perfect 
> for an LTS cycle. I personally think we should go through all the 
> groups during regular releases as well, but possibly group more groups 
> into one call, and relax on the amount of testing "required". Optional 
> tests could be literally that; run if comfortable, but if they are 
> left untested, that's fine as well.
>
> As to what (else) to test, I think we should try to focus on new 
> features, as we did this cycle. This can and probably should be 
> extended to running tests on applications that have had a major update 
> during the cycle. All of this in a flexible manner; the more new 
> things we have about to test, the looser running the other tests 
> should be. Except on the LTS releases...
>
> I've yet to decide if some of the testcases are a bit too thorough or 
> if they are just about right. I guess we can agree and assume that the 
> amount of bugs is somewhat correlating with how deep the tests are. As 
> I see it though, the deeper and specific the tests are, the more 
> mechanic running them is. Which leads us to exploratory testing...
>
> I have a few doubtful thoughts on exploratory testing. How do we 
> motivate people to run exploratory testing with the development 
> version, while it is not ready for production, or day-to-day 
> environments? If the tests aren't run on/as your main system, how can 
> the testing be natural enough to be of exploratory nature? How do we 
> specify a good balance between feature and exploratory testing?
>
> MILESTONE (ISO) TESTING
>
> It is hard to evaluate how the milestone ISO testing succeeded because 
> we still have one beta to go, which is also the most important 
> milestone. That is something where we can improve though.
>
> The alpha releases could have been focused more on specific issues. 
> Now we kind of just ran through them without clear focus. Of course 
> this means that developers need to have their stuff together earlier 
> in the cycle, but that is a desirable direction generally.
>
> I would rethink the amount of alpha releases we want to participate in 
> especially with non-LTS releases. We can opt-in for as many as we did 
> now if we have set a clear point of focus for those. This looks 
> unrealistic for T+1 though, as this cycle has been really busy for 
> everybody and we have got a lot of stuff that was prepared in the last 
> 2 years included.
>
> For the beta releases, we should get more publicity. We still have the 
> beta 2 release to come, so let's try to fix at least some of that for 
> Trusty.
>
> CONCLUSION
>
> To end the feedback on a positive note (though there weren't so many 
> negative points in total anyway), I think we have been up to the 
> highest possible standard with QA considering the size of our team and 
> the amount of new things landing this cycle.
>
> Finally, a big THANK YOU Elfy for running the QA team, doing all the 
> calls, reporting back to us, taking care of bugs being noticed, 
> features landing in time et cetera... Last but not least, thanks for 
> putting up with us all who have sometimes more or less neglected our 
> duties in QA and being unresponsive to questions and calls. It is very 
> much appreciated, and I totally think that 14.04 would be a lesser 
> release without your work and persistence!
>
> Cheers,
> Pasi
>
Rather than post to the last mail I'll reply to this one.

Thanks for the feedback by everyone - much appreciated :)

So I've taken this from the comments.

    *Testcase grouping* - call for more than one at a time, I'll likely
    be re-organising some of them post 14.04 as well.
    *
    **Optional testcases* - can leave these for non-LTS testing

    *New feature testing* - much as we did this cycle, fit them in when
    we can - existing testcases to take a back seat if new features need
    testing.

    *Exploratory testing* - I'm not looking at this any longer - or at
    least, it needs to work in conjunction with autopilot testing, there
    will be a mail to the list in the near future about this from one of
    the other members of the QA team.

    *Specific Testing during milestones* - Work specific package testing
    into various milestones when it's appropriate for us. Necessarily
    this will need to be led by devs - they'll know more about what
    needs to be tested. Only take part in milestones when there is a need.

    *Testcase feedback* - I'll send a mail to the list regarding this
    seperately, those that have actually taken part in package testing -
    your input on this will be invaluable, please join in with this
    discussion.

    *Feedback* to the list does help us - but it is a whole lot easier
    to follow the trackers, bugs reported to those end up on our
    blueprints during cycles - we can track that. Mailing list threads -
    not trackable. In addition when you are reporting to a tracker it
    will tell you bugs that others have reported against that test, be
    it a package or an image.

Elfy

-- 
Ubuntu Forum Council Member
Xubuntu QA Lead

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/xubuntu-devel/attachments/20140327/fa5afbaa/attachment.html>