Feedback on the QA cycle

[Elfy]

On 20/03/14 23:53, Elizabeth Krumbach Joseph wrote:
> On Thu, Mar 20, 2014 at 4:38 PM, Pasi Lallinaho <pasi at shimmerproject.org> wrote:
>> this is a reply to the QA recap/feedback thread. As the original thread went
>> off track, I decided to start a new one to discuss the original question at
>> hand.
> Thanks, I do hope we can keep this one on track as it's an important discussion
>
>> PACKAGE TESTING
>>
>> First of all, I think it was a good move to run the package testing in
>> groups and in cadence before we hit the beta milestones. Running all those
>> tests and gathering a (big) list of bugs was and is important, especially
>> now that we have entered the "bug fixes only" stage of the release
>> preparing. I am sure we would be able to fix a lot less bugs that are
>> annoying and affect numerous of people.
>>
>> That being said, I think the amount of calls was just about perfect for an
>> LTS cycle. I personally think we should go through all the groups during
>> regular releases as well, but possibly group more groups into one call, and
>> relax on the amount of testing "required". Optional tests could be literally
>> that; run if comfortable, but if they are left untested, that's fine as
>> well.
> As far as the calls for testing go, both for packages (even though I
> didn't participate, sorry!) and ISOs (where I did!), I found the
> direct emails via Launchpad super helpful (they go to my Inbox, not
> filtered to -devel). It also really made me feel like I knew what was
> going on with testing so when others asked me how they could jump in I
> had a good starting point to know where the help was required.
>

This is useful to know - thanks :)
>> As to what (else) to test, I think we should try to focus on new features,
>> as we did this cycle. This can and probably should be extended to running
>> tests on applications that have had a major update during the cycle. All of
>> this in a flexible manner; the more new things we have about to test, the
>> looser running the other tests should be. Except on the LTS releases...
>>
>> I've yet to decide if some of the testcases are a bit too thorough or if
>> they are just about right. I guess we can agree and assume that the amount
>> of bugs is somewhat correlating with how deep the tests are. As I see it
>> though, the deeper and specific the tests are, the more mechanic running
>> them is. Which leads us to exploratory testing...
> I'd love to hear some feedback about the thoroughness of the
> testcases. We don't want folks to be put off when they see the test
> case being long, but at the same time it won't have much value if it's
> short.
I too would love to get feedback on testcases - but I would much rather 
that came as specific mails to the list, rather than lumped in with this.


>> I have a few doubtful thoughts on exploratory testing. How do we motivate
>> people to run exploratory testing with the development version, while it is
>> not ready for production, or day-to-day environments? If the tests aren't
>> run on/as your main system, how can the testing be natural enough to be of
>> exploratory nature? How do we specify a good balance between feature and
>> exploratory testing?
> I think what we'd struggle with is not people being unwilling to do
> the testing, as we know there are lots of people who do actually run
> development versions since we're always hearing feedback about how
> stable it is :) I think the issue is connecting them with bug
> reporting and other mechanisms for reporting results. I think if we
> even got feedback given via the mailing list it would be helpful. Not
> sure how to make this easier for people.

I'm not sure that having a launchpad account and going to the tracker 
could be much easier. While people's mails to the list do get read. It's 
not so easy to follow up. A bug reported via the tracker ticks all the 
boxes.
>> MILESTONE (ISO) TESTING
>>
>> It is hard to evaluate how the milestone ISO testing succeeded because we
>> still have one beta to go, which is also the most important milestone. That
>> is something where we can improve though.
>>
>> The alpha releases could have been focused more on specific issues. Now we
>> kind of just ran through them without clear focus. Of course this means that
>> developers need to have their stuff together earlier in the cycle, but that
>> is a desirable direction generally.
>>
>> I would rethink the amount of alpha releases we want to participate in
>> especially with non-LTS releases. We can opt-in for as many as we did now if
>> we have set a clear point of focus for those. This looks unrealistic for T+1
>> though, as this cycle has been really busy for everybody and we have got a
>> lot of stuff that was prepared in the last 2 years included.
>>
>> For the beta releases, we should get more publicity. We still have the beta
>> 2 release to come, so let's try to fix at least some of that for Trusty.
> I could have done a much better job at handling social media for this
> cycle, I should discipline myself to send something out whenever a
> call hits my Inbox... or add more admins on social media to help
> handle this.
This could also be helped by -QA people pinging you in channel :)
>> CONCLUSION
>>
>> To end the feedback on a positive note (though there weren't so many
>> negative points in total anyway), I think we have been up to the highest
>> possible standard with QA considering the size of our team and the amount of
>> new things landing this cycle.
>>
>> Finally, a big THANK YOU Elfy for running the QA team, doing all the calls,
>> reporting back to us, taking care of bugs being noticed, features landing in
>> time et cetera... Last but not least, thanks for putting up with us all who
>> have sometimes more or less neglected our duties in QA and being
>> unresponsive to questions and calls. It is very much appreciated, and I
>> totally think that 14.04 would be a lesser release without your work and
>> persistence!
> Absolutely, Elfy's really done an exceptional job staying on top of
> all of this even with all his other commitments to Ubuntu and beyond.
> Thank you for your work!
>


-- 
Ubuntu Forum Council Member
Xubuntu QA Lead