bzr maverick srus now qa'd

[Martin Pool]

Elsewhere, on 26 April 2011 08:13, Martin Pitt <martin.pitt at ubuntu.com> wrote:
> Hello Martin,
>
> Martin Pool [2011-04-21 18:48 +1000]:
>> On the whole I'm not sure [manually verifying all SRU bugs in bzr] was a good use of time: I think we tend
>> to have problems not so much when we fail to fix the bug but rather
>> when we break something else in doing so.  Manually testing the bug is
>> probably not going to catch that, and is probably redundant with
>> testing we did and the original reporter did when it was merged
>> upstream.
>
> I agree. Indeed I'm much more concerned about regression testing,
> which is of course hard to describe in general for all SRUs. So we
> usually resort to minimal patches and have reporters test the actual
> package in a real environment. Strictly speaking this is not true
> regression testing, but the next best thing to what we can reasonably
> achieve.
>
> In the bzr case however, we can do proper regression testing, because
> it already has a huge test suite. Indeed the MRE says:
>
>  conditions: test suite running during package build from Ubuntu
>  11.04 on; SRU verification should run test suite in installed sytem
>
> From my POV doing the latter and then reporting back to one of the
> bugs with "test suite showed no regressions for the package in
> foo-proposed" would suffice here. I think that was the original intent
> when we discussed the MRE.
>
> We handle things in a similar way for other MREs like postgresql or
> Firefox: We run standard tests (manual or automatic suites) for
> signing them off.

I would like to ask the tech board to approve an edit
<https://wiki.ubuntu.com/StableReleaseUpdates/MicroReleaseExceptions>
to say something along these lines, in the interests of people getting
changes without wasted effort or inconsistent handling.

Thanks
Martin