Feedback on the QA cycle

Fri Mar 21 00:05:50 UTC 2014

On 2014-03-21 01:53, Elizabeth Krumbach Joseph wrote:
> On Thu, Mar 20, 2014 at 4:38 PM, Pasi Lallinaho <pasi at shimmerproject.org> wrote:
>> this is a reply to the QA recap/feedback thread. As the original thread went
>> off track, I decided to start a new one to discuss the original question at
>> hand.
> Thanks, I do hope we can keep this one on track as it's an important discussion
>
>> PACKAGE TESTING
>>
>> First of all, I think it was a good move to run the package testing in
>> groups and in cadence before we hit the beta milestones. Running all those
>> tests and gathering a (big) list of bugs was and is important, especially
>> now that we have entered the "bug fixes only" stage of the release
>> preparing. I am sure we would be able to fix a lot less bugs that are
>> annoying and affect numerous of people.
>>
>> That being said, I think the amount of calls was just about perfect for an
>> LTS cycle. I personally think we should go through all the groups during
>> regular releases as well, but possibly group more groups into one call, and
>> relax on the amount of testing "required". Optional tests could be literally
>> that; run if comfortable, but if they are left untested, that's fine as
>> well.
> As far as the calls for testing go, both for packages (even though I
> didn't participate, sorry!) and ISOs (where I did!), I found the
> direct emails via Launchpad super helpful (they go to my Inbox, not
> filtered to -devel). It also really made me feel like I knew what was
> going on with testing so when others asked me how they could jump in I
> had a good starting point to know where the help was required.

If people think that this is a good thing, then great to see the 
~xubuntu-testers team to have some use. At the same time I think it's a 
good change that people actually expire from the team (unless they 
renew) to keep some track of the potential tester amount.

>> As to what (else) to test, I think we should try to focus on new features,
>> as we did this cycle. This can and probably should be extended to running
>> tests on applications that have had a major update during the cycle. All of
>> this in a flexible manner; the more new things we have about to test, the
>> looser running the other tests should be. Except on the LTS releases...
>>
>> I've yet to decide if some of the testcases are a bit too thorough or if
>> they are just about right. I guess we can agree and assume that the amount
>> of bugs is somewhat correlating with how deep the tests are. As I see it
>> though, the deeper and specific the tests are, the more mechanic running
>> them is. Which leads us to exploratory testing...
> I'd love to hear some feedback about the thoroughness of the
> testcases. We don't want folks to be put off when they see the test
> case being long, but at the same time it won't have much value if it's
> short.
>
>> I have a few doubtful thoughts on exploratory testing. How do we motivate
>> people to run exploratory testing with the development version, while it is
>> not ready for production, or day-to-day environments? If the tests aren't
>> run on/as your main system, how can the testing be natural enough to be of
>> exploratory nature? How do we specify a good balance between feature and
>> exploratory testing?
> I think what we'd struggle with is not people being unwilling to do
> the testing, as we know there are lots of people who do actually run
> development versions since we're always hearing feedback about how
> stable it is :) I think the issue is connecting them with bug
> reporting and other mechanisms for reporting results. I think if we
> even got feedback given via the mailing list it would be helpful. Not
> sure how to make this easier for people.

That's a valid concern and a good point. Can anybody come up with ideas 
on how we can help the people "reporting" their results to convert them 
to bug reports, or measureable reports? Feedback from people who have 
sent testing reports to this mailing list but not touched the QA 
trackers would be more than welcome.

We have brainstormed over writing some developer/contributor 
documentation, would this be a good start for that? If we wrote some 
documentation on this, where would the target audience wish to see this? 
I would imagine online, but would it be in the team blog, as a page on 
the website, in the developer wiki or possibly in the documentation area 
under docs.xubuntu.org?

>> MILESTONE (ISO) TESTING
>>
>> It is hard to evaluate how the milestone ISO testing succeeded because we
>> still have one beta to go, which is also the most important milestone. That
>> is something where we can improve though.
>>
>> The alpha releases could have been focused more on specific issues. Now we
>> kind of just ran through them without clear focus. Of course this means that
>> developers need to have their stuff together earlier in the cycle, but that
>> is a desirable direction generally.
>>
>> I would rethink the amount of alpha releases we want to participate in
>> especially with non-LTS releases. We can opt-in for as many as we did now if
>> we have set a clear point of focus for those. This looks unrealistic for T+1
>> though, as this cycle has been really busy for everybody and we have got a
>> lot of stuff that was prepared in the last 2 years included.
>>
>> For the beta releases, we should get more publicity. We still have the beta
>> 2 release to come, so let's try to fix at least some of that for Trusty.
> I could have done a much better job at handling social media for this
> cycle, I should discipline myself to send something out whenever a
> call hits my Inbox... or add more admins on social media to help
> handle this.
>
>> CONCLUSION
>>
>> To end the feedback on a positive note (though there weren't so many
>> negative points in total anyway), I think we have been up to the highest
>> possible standard with QA considering the size of our team and the amount of
>> new things landing this cycle.
>>
>> Finally, a big THANK YOU Elfy for running the QA team, doing all the calls,
>> reporting back to us, taking care of bugs being noticed, features landing in
>> time et cetera... Last but not least, thanks for putting up with us all who
>> have sometimes more or less neglected our duties in QA and being
>> unresponsive to questions and calls. It is very much appreciated, and I
>> totally think that 14.04 would be a lesser release without your work and
>> persistence!
> Absolutely, Elfy's really done an exceptional job staying on top of
> all of this even with all his other commitments to Ubuntu and beyond.
> Thank you for your work!
>
Cheers,
Pasi

-- 
Pasi Lallinaho (knome)                      » http://open.knome.fi/
Leader of Shimmer Project and Xubuntu       » http://shimmerproject.org/
Graphic artist, webdesigner, Ubuntu member  » http://xubuntu.org/