Release Process concerns (QA) and suggestions

Fri Aug 31 10:38:29 UTC 2012

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Stéphane,

thanks for your answer, it gives us many clues to improve coverage at
milestones, which is good news.

On 30/08/12 19:53, Stéphane Graber wrote:
> On 08/28/2012 09:38 AM, Gema Gomez wrote: I'm going to start by 
> saying that we never respin for fun, when we respin, there's a 
> really good reason to do so.

I didn't want to imply you respin for fun, I'd just like to know the
reasons to be able to preempt the respins and find problems as early
as possible.

> The release team is in charge of releasing a pre-defined set of 
> images, for a given list of media at a given date. That's how 
> things are.
> 
> When we unfortunately hit a bug at the last minute, like happened 
> last week, the release team needs to check how critical it's. If 
> it's considered as a show-stopper, like was the case here, the
> only action to take is to fix it as soon as possible, re-test and
> then release.
> 
> If we know it's technically impossible to get it re-tested in
> time, then we need to release a day later, but that's a very last
> resort as releasing on a Friday brings its own set of problems.

Which kind of problems do you face when releasing on Friday? I think
it'd be good for us to know the consequences as well.

> In the case of 12.04.1, we noticed on release day that an image 
> didn't actually fit on its target media and apparently no tester 
> bothered to actually burn it to a standard CD...

You could use du next time right after the image is built to satisfy
yourself that the size is good, it could be a standard check that you
guys do. We have added some static validation tests to jenkins and are
in the process of publishing them. I don't think we need to burn a CD
to know if the image is going to fit or not. But if you want us to
validate things manually, adding a test case to the current set in the
iso tracker will help track that someone has bothered. Unfortunately I
don't feel confident enough yet with the admin mode of the iso tracker
to change anything, so your expertise there would be appreciated.

Anyway, looking forward rather than backwards, for Quantal the size is
800MB so, what media do you suggest we test on for size next week?

https://blueprints.launchpad.net/ubuntu/+spec/desktop-q-one-iso-for-q

> We found an obvious way of fixing it (removing a langpack) within 
> just a couple of hours, got the change reviewed, tested, the image 
> rebuilt, the content checked and then fully re-tested by 3 testers 
> in less than 3 hours. Leaving us a good 10-12h before we actually 
> released the set.

In my opinion, it is not possible for 3 people to do 10 installs + 3
upgrades each to a good level of details in less than 3 hours. Yes,
you can rush through things or split the test cases between the three
of you, or consider some tests done because one test case is sort of a
subset of another, and do some risk based testing, but the level of
risk we are accepting is not clear nor understood by all the parties.

> So we had more peer review than required at every step, I'm really 
> not understanding what you're complaining about.

See above for explanation. We clearly have different views on what is
required/acceptable, and we need to reach an agreement, something that
works for all of us.

> 
> We always respin responsibly, believe it or not, respinning is at 
> least as much pain for the release team as it's for the testers,
> we never take such action lightly.

I believe it, I'd like to reduce the pain for everyone.

> Critical installation bugs, security bugs, immediate post-install 
> bugs and CD size problems are usually considered show-stoppers as 
> these can't be worked around by the user. It'd be wrong not to 
> respin for these.

This is good information. What do you mean by usually? do you mean
always? What would be an example case of those showstoppers that
doesn't grant a respin?

> 
> Obviously I'm also dreaming of a world where software doesn't have 
> bug or where issues are found months ahead of release, but sadly 
> that world doesn't exist and we have to live with reality.

Indeed, but there are many shades for reality, we just need to find
one that works for everyone. As I said, quality is not about releasing
without bugs, it is about knowing what level of quality we are releasing.

> The "corner case" in [4] is a supported upgrade path, used by 
> governments and other internet-less environments. Not fixing that 
> bug was resulting in completely broken, unbootable systems, and as 
> such definitely fits respin criteria. Our alternative would have 
> been to drop support for these, which we considered and decided
> not to for 12.04. However 12.04 is going to be the last release
> where such an upgrade path is supported.

Useful information, thanks, we will add this test case earlier in the
testing next time, so that we don't find ourselves in that situation
again. Does this only apply to LTSs or does it also apply to
intermediate releases? Are such customers be likely to upgrade to
releases in between LTSs?

> We have a limited set of people from the release team who are in 
> charge of spinning images, these are the ones making the decision.
> 
> You can see who's in charge of what here: 
> https://wiki.ubuntu.com/QuantalQuetzal/ReleaseTaskSignup

Thanks, this is useful information!

> I suppose we can do that, ultimately it's always going to be up to 
> the release team to do a go/no-go on case by case basis, but 
> writting some generic guidelines can't hurt.
> 
> At least for me, anything that fits one of the following is release
> critical: - Security issues affecting the live/install environment
> - Kernel bugs preventing the boot of the image for commonly
> available hardware - Installer bugs leading to installation failure
> or broken post-install experience without obvious workaround -
> Upgrade bugs leading to broken/non-working systems that can't be
> fixed post-upgrade through SRU. - Critical bugs affecting common
> software used immediately post-installation

This is a good starting point, and will help us focus our testing
going forward. Any other show-stopper kind of issue you can think of?
Or someone else?

>> - Let's improve the static analysis of images so that we don't 
>> have the image size problem again, we are adding a job for this 
>> to Jenkins this week.
> 
> Can you also make sure someone actually burns the image on the 
> supported media?

As I asked before, what media is the preferred one for the 800MB
Quantal images? We'll be happy to procure those and make sure we burn
it. I'd like to be able to track this has been done with a test case
on the tracker.

The static validation is added to jenkins and results will start to be
published to the public instance today.

> I'm still amazed that for a whole week, nobody even tried to burn
> a CD with our image...

I am amazed that you expect things to happen without actually having
this documented anywhere, we don't have a test case for this. This is
a problem for us, because the team is growing and not everyone has the
years of experience that jibel has, nor has seen things fail in so
many different ways as to use intuition. Jibel is trying to move on to
do upstream testing, so it is our responsibility as a team to be able
to do a good job and we need transparency from the release team to
achieve that.

As I said in a different email, jibel won't be the QA single point of
contact anymore, plars will be leading the milestone testing efforts
for the last milestones in Quantal with psivaa and babyface's help,
and then we will work out a schedule or a plan for R that we will
communicate in due time, so that we are all clear what is going
to be tested and can tell us if anything is missing.

This means that for new testers to be able to reach his level of
competence, we need to suffer because things are not documented. We
are learning from the mistakes and will try not to make them again,
but we've reached a point where trusting a single person to remember
everything is not practical.

>> - Let's require more than just one run of the test cases to 
>> validate an image. What is reasonable in terms of ensuring 
>> reasonable HW coverage? I'd like to see at least 3 x 100% run 
>> rate with 100% pass rate on the current test cases, from people 
>> different from the release engineer.
> 
> For final images I usually look for at least 2 people testing the 
> various code paths. Unfortunately these code paths can't be easily
>  represented in the UI, so ultimately it's a release team decision 
> to know whether the threshold has been met.

I'd like someone from QA at least involved in the decision process,
even if only as an assessor, to voice our concerns.

> Sorry for the rather long e-mail, but I hope it's explaining a bit 
> more how things work.

It is very helpful, thanks. This email contains information we can use
to change things preemptively rather than reactively, like we've done
in the past.

> As I said on IRC and discussed with the rest of the team, I don't 
> believe anything was done wrong, I do believe our images were 
> sufficiently tested and as far as I know, nothing actually went 
> wrong.

Maybe nothing has been done wrongly, but it's good to look for ways of
improving, that's my take.

> You don't seem to agree and I can acknowledge that, however I don't
> intend on changing anything and I don't think we should have.

I feel the pain of things not being transparent, but at the end of the
day, no big problems occurred, so that's a good sign. That doesn't
mean we cannot work out a smoother process for everyone's benefit.

Thanks for your feedback, much appreciated!
Gema

- -- 
Gema Gomez-Solano        <gema.gomez-solano at canonical.com>
Ubuntu QA Team           https://launchpad.net/~gema.gomez
Canonical Ltd.           http://www.canonical.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQQJQfAAoJEGrsYwHIaYlgFKAH/iqFdbjos/1MDQQjQrKfpftU
Q80Y8oII0fWRtU1uBFwUCmwy9SDCfXHHdBzOX4BH9gvWVdCFivFykNULEqZk7Az4
SQ/O8JTcKJpMeOI8M5/qofchzhyhaVCP4rxkK/mU073E2AnekVKNB74wrlb/jvfV
q32AZmYWxxqsR/CVnqbQpP9oDinfkB5t6zpyWCDTaHGm3Bwgl0tWgc7YrvVg0G+6
ZIe6deBe/qtYzu/IMiXlkMGnxGxTPc5cndOaT2FWhYuBH3kw8qXVl8S1aqfKiT44
l+w2DWFKrgDb1fF2Rayt2b7212A3HQ3njLBlnKsB57CCg5zgZKQEmYVMucERjYg=
=AHOw
-----END PGP SIGNATURE-----