Feedback on the QA cycle

[Elizabeth Krumbach Joseph]

On Thu, Mar 20, 2014 at 4:38 PM, Pasi Lallinaho <pasi at shimmerproject.org> wrote:
> this is a reply to the QA recap/feedback thread. As the original thread went
> off track, I decided to start a new one to discuss the original question at
> hand.

Thanks, I do hope we can keep this one on track as it's an important discussion

> PACKAGE TESTING
>
> First of all, I think it was a good move to run the package testing in
> groups and in cadence before we hit the beta milestones. Running all those
> tests and gathering a (big) list of bugs was and is important, especially
> now that we have entered the "bug fixes only" stage of the release
> preparing. I am sure we would be able to fix a lot less bugs that are
> annoying and affect numerous of people.
>
> That being said, I think the amount of calls was just about perfect for an
> LTS cycle. I personally think we should go through all the groups during
> regular releases as well, but possibly group more groups into one call, and
> relax on the amount of testing "required". Optional tests could be literally
> that; run if comfortable, but if they are left untested, that's fine as
> well.

As far as the calls for testing go, both for packages (even though I
didn't participate, sorry!) and ISOs (where I did!), I found the
direct emails via Launchpad super helpful (they go to my Inbox, not
filtered to -devel). It also really made me feel like I knew what was
going on with testing so when others asked me how they could jump in I
had a good starting point to know where the help was required.

> As to what (else) to test, I think we should try to focus on new features,
> as we did this cycle. This can and probably should be extended to running
> tests on applications that have had a major update during the cycle. All of
> this in a flexible manner; the more new things we have about to test, the
> looser running the other tests should be. Except on the LTS releases...
>
> I've yet to decide if some of the testcases are a bit too thorough or if
> they are just about right. I guess we can agree and assume that the amount
> of bugs is somewhat correlating with how deep the tests are. As I see it
> though, the deeper and specific the tests are, the more mechanic running
> them is. Which leads us to exploratory testing...

I'd love to hear some feedback about the thoroughness of the
testcases. We don't want folks to be put off when they see the test
case being long, but at the same time it won't have much value if it's
short.

> I have a few doubtful thoughts on exploratory testing. How do we motivate
> people to run exploratory testing with the development version, while it is
> not ready for production, or day-to-day environments? If the tests aren't
> run on/as your main system, how can the testing be natural enough to be of
> exploratory nature? How do we specify a good balance between feature and
> exploratory testing?

I think what we'd struggle with is not people being unwilling to do
the testing, as we know there are lots of people who do actually run
development versions since we're always hearing feedback about how
stable it is :) I think the issue is connecting them with bug
reporting and other mechanisms for reporting results. I think if we
even got feedback given via the mailing list it would be helpful. Not
sure how to make this easier for people.

> MILESTONE (ISO) TESTING
>
> It is hard to evaluate how the milestone ISO testing succeeded because we
> still have one beta to go, which is also the most important milestone. That
> is something where we can improve though.
>
> The alpha releases could have been focused more on specific issues. Now we
> kind of just ran through them without clear focus. Of course this means that
> developers need to have their stuff together earlier in the cycle, but that
> is a desirable direction generally.
>
> I would rethink the amount of alpha releases we want to participate in
> especially with non-LTS releases. We can opt-in for as many as we did now if
> we have set a clear point of focus for those. This looks unrealistic for T+1
> though, as this cycle has been really busy for everybody and we have got a
> lot of stuff that was prepared in the last 2 years included.
>
> For the beta releases, we should get more publicity. We still have the beta
> 2 release to come, so let's try to fix at least some of that for Trusty.

I could have done a much better job at handling social media for this
cycle, I should discipline myself to send something out whenever a
call hits my Inbox... or add more admins on social media to help
handle this.

> CONCLUSION
>
> To end the feedback on a positive note (though there weren't so many
> negative points in total anyway), I think we have been up to the highest
> possible standard with QA considering the size of our team and the amount of
> new things landing this cycle.
>
> Finally, a big THANK YOU Elfy for running the QA team, doing all the calls,
> reporting back to us, taking care of bugs being noticed, features landing in
> time et cetera... Last but not least, thanks for putting up with us all who
> have sometimes more or less neglected our duties in QA and being
> unresponsive to questions and calls. It is very much appreciated, and I
> totally think that 14.04 would be a lesser release without your work and
> persistence!

Absolutely, Elfy's really done an exceptional job staying on top of
all of this even with all his other commitments to Ubuntu and beyond.
Thank you for your work!

-- 
Elizabeth Krumbach Joseph || Lyz || pleia2
http://www.princessleia.com