Proposed pocket racing uninstallability and SRU verification around release time

[Jason DeRose]

On 10/17/2016 01:55 PM, Steve Langasek wrote:
> Hi Robie,
>
> [corrected the launchpad bug cc]
>
> On Mon, Oct 17, 2016 at 05:57:07PM +0100, Robie Basak wrote:
>> I just filed this bug: https://bugs.launchpad.net/ubuntu/+bug/1634201; I
>> Cc'd the bug so as to try and not fragment any discussion.
>
>> During development, we have packages in -proposed that fail to migrate,
>> as expected, for good reason.
>
>> At release time, these packages are still present. For example, Yakkety
>> released with libhcrypto4-heimdal:amd64 1.7~git20160703+dfsg-1 in
>> proposed, which is not in the release pocket because it is broken (see
>> bug 1617963).
>
>> I think we should be encouraging users to volunteer to risk testing
>> proposed in stable releases. This helps with SRU verification.
>
>> However, our current release process breaks these users when they
>> upgrade to a new release (which, given that they are testing the cutting
>> edge, they are likely to do early, before the proposed pocket has been
>> cleaned out).
>
>> This means that users, instead of being encouraged, are being
>> discouraged from testing the SRU proposed pocket since we are breaking
>> them with known bugs but delaying removal of those breakages.
>
>> Bug 1633653 is an example: a user with xenial-proposed enabled upgraded
>> to Yakkety one day after release, and this broke.
>
>> How can we adjust our release process to stop this happening?
>
> I have previously argued that -proposed should be treated the same way as
> -backports on end user systems, allowing users to have it enabled and select
> specific packages for installation without an apt upgrade pulling everything
> from that pocket.  While the problem you describe is /worst/ right after a
> release has been marked stable, it's not unique to that time of the cycle;
> -proposed is always the place where packages sit *before* they are
> guaranteed to be coherently installable, so a wholesale upgrade is always
> risky.  Just as an example, we have had various bug reports in the past
> about users who have enabled -proposed, then rendered their systems
> unbootable under Secure Boot because they upgraded at just the right moment
> for grub-efi-amd64-signed to come uninstalled.  This is intrinsic to the
> core function of -proposed, and not something that we can shield users from
> except by discouraging them from dist-upgrading to -proposed.
>
> I don't think end users should be installing random packages from -proposed
> for purposes of fuzz testing of SRUs; it's just not worth the collateral
> damage nowadays.

Steve,

While I agree that *most* users shouldn't be bothering with -proposed, 
there is still an important question: is the intent of -propsed to 
reflect the likely soon-to-be state of a stable release via SRUs?

I think I have a good use-case for this: System76 would like to be much 
more aggressive about testing -proposed to prevent regressions from 
reaching our customers. In this case, it's not a matter of testing a 
specific package in -proposed, it's a matter of testing *all* packages 
in -proposed (and their interactions)... for the purpose of testing the 
expected state of a stable release in the near future.

I think the concern that Robie Basak originally brought up (please 
correct me if I'm wrong, Robbie) is that -proposed so often is littered 
with packaging that will most likely *never* make their way into 
-updates for a stable release. Yes, this tends to be worse during the 
dev cycle for a new stable release and immediately after its release, 
but it can also be quite bad during the lifetime of an LTS release.

For example, if you look at:

http://people.canonical.com/~ubuntu-archive/pending-sru.html

Xenial already has a package in -proposed that has been sitting there 
for 138 days (librarian-puppet-simple). Grated, this package isn't 
installed by default and so understandably is less critical in the grand 
scheme of things. But previous experience watching this SRU page 
suggests to me that this proposed `librarian-puppet-simple` package will 
*never* make it to xenial-updates... so why is this package still there?

For a while I've thought that perhaps a reasonable solution would be to 
automatically remove -proposed packages that fail to migrate to -updates 
within a certain amount of time (one month, two months, whatever it may be).

I guess in summary, I see two important use cases for -proposed:

1) For stakeholders to test a single specific (source) package in 
-proposed, especially for SRU verification when said stake holder either 
filed a specific bug or is effected by a specific bug

2) For stakeholders to test, in total across all -proposed packages, the 
soon-to-be expected state of the stable release, without awareness of 
the details of any of these -proposed packages (especially without 
needing to guess which specific ones are test-worthy)

I think making -proposed work like -backports can totally fix use case 
(1), but I don't see how it really helps use case (2).

So is there any modest incremental step that can be taken to make 
-proposed better reflect the likely total soon-to-be state of the stable 
release, to eliminate the noise from packages that are already past the 
point of likely ever reaching -updates?

My 2 cents :)
-Jason