Feedback on the QA cycle

[David Pires]

>
> Date: Fri, 21 Mar 2014 01:38:49 +0200
> From: Pasi Lallinaho <pasi at shimmerproject.org>
> To: Xubuntu Development <xubuntu-devel at lists.ubuntu.com>
> Subject: Feedback on the QA cycle
> Message-ID: <532B7C09.9050207 at shimmerproject.org>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hello,
>
> this is a reply to the QA recap/feedback thread. As the original thread
> went off track, I decided to start a new one to discuss the original
> question at hand.
>

Good call.


> PACKAGE TESTING
>
> First of all, I think it was a good move to run the package testing in
> groups and in cadence before we hit the beta milestones. Running all
> those tests and gathering a (big) list of bugs was and is important,
> especially now that we have entered the "bug fixes only" stage of the
> release preparing. I am sure we would be able to fix a lot less bugs
> that are annoying and affect numerous of people.
>

I share this view. Just see the number of bug listed in
https://blueprints.launchpad.net/ubuntu/+spec/xubuntu-t-bugs which proves
exactly that.

That being said, I think the amount of calls was just about perfect for
> an LTS cycle. I personally think we should go through all the groups
> during regular releases as well, but possibly group more groups into one
> call, and relax on the amount of testing "required". Optional tests
> could be literally that; run if comfortable, but if they are left
> untested, that's fine as well.
>
> As to what (else) to test, I think we should try to focus on new
> features, as we did this cycle. This can and probably should be extended
> to running tests on applications that have had a major update during the
> cycle. All of this in a flexible manner; the more new things we have
> about to test, the looser running the other tests should be. Except on
> the LTS releases...
>

+1


> I've yet to decide if some of the testcases are a bit too thorough or if
> they are just about right. I guess we can agree and assume that the
> amount of bugs is somewhat correlating with how deep the tests are. As I
> see it though, the deeper and specific the tests are, the more mechanic
> running them is. Which leads us to exploratory testing...
>

I agree that some of the tests may seem excessively long, and perhaps even
dense. But, and even falling in the role of advocating my own cause since
it was I who wrote some of the longer tests, I believe that in some
specific programs the existence of more complex and demanding tests is
justifiable in what QA is concerned and aims.

I have a few doubtful thoughts on exploratory testing. How do we
> motivate people to run exploratory testing with the development version,
> while it is not ready for production, or day-to-day environments? If the
> tests aren't run on/as your main system, how can the testing be natural
> enough to be of exploratory nature? How do we specify a good balance
> between feature and exploratory testing?
>

People tend to be afraid of running exploratory tests on their production
systems, which is perfectly understandable, and most of this exploratory
tests will for sure be made in virtual machines in 90% of the cases.
Even though theoretically speaking, applications running on a virtual
machine behave as if they were running on their own physical system, we all
know that's not the same thing as running them in real hardware
environments, particularly when we speak of OS testing.

IMO, a reliable option would be the use of a automated testing tool, such
as Autopilot, extending features for testing applications such as unit
testing, regression testing, GUI testing, Web testing, distributed testing
and many others.

MILESTONE (ISO) TESTING
>
> It is hard to evaluate how the milestone ISO testing succeeded because
> we still have one beta to go, which is also the most important
> milestone. That is something where we can improve though.
>
> The alpha releases could have been focused more on specific issues. Now
> we kind of just ran through them without clear focus. Of course this
> means that developers need to have their stuff together earlier in the
> cycle, but that is a desirable direction generally.
>
> I would rethink the amount of alpha releases we want to participate in
> especially with non-LTS releases. We can opt-in for as many as we did
> now if we have set a clear point of focus for those. This looks
> unrealistic for T+1 though, as this cycle has been really busy for
> everybody and we have got a lot of stuff that was prepared in the last 2
> years included.
>
> For the beta releases, we should get more publicity. We still have the
> beta 2 release to come, so let's try to fix at least some of that for
> Trusty.
>

That pretty much sums it all. Adding up the pros and cons of this cycle so
far, I'm under the impression that we manage to come out with a positive
balance.

CONCLUSION
>
> To end the feedback on a positive note (though there weren't so many
> negative points in total anyway), I think we have been up to the highest
> possible standard with QA considering the size of our team and the
> amount of new things landing this cycle.
>
> Finally, a big THANK YOU Elfy for running the QA team, doing all the
> calls, reporting back to us, taking care of bugs being noticed, features
> landing in time et cetera... Last but not least, thanks for putting up
> with us all who have sometimes more or less neglected our duties in QA
> and being unresponsive to questions and calls. It is very much
> appreciated, and I totally think that 14.04 would be a lesser release
> without your work and persistence!
>

I can't +1 enough this.
Elfy, thanks for being such a persistent and tireless marathon runner.

Cheers,
David Pires
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/xubuntu-devel/attachments/20140321/8d94612a/attachment.html>