Proposed pocket racing uninstallability and SRU verification around release time

[Steve Langasek]

Hi Jason,

On Wed, Oct 19, 2016 at 10:37:25AM -0600, Jason DeRose wrote:
> >I don't think end users should be installing random packages from -proposed
> >for purposes of fuzz testing of SRUs; it's just not worth the collateral
> >damage nowadays.

> While I agree that *most* users shouldn't be bothering with -proposed, there
> is still an important question: is the intent of -propsed to reflect the
> likely soon-to-be state of a stable release via SRUs?

Well, as long as you filter out packages that:

 - have not yet built on all architectures
 - are out of date on one or more architectures because they have failed to
   build
 - have one or more SRU bugs marked verification-failed
 - have undeclared / unrepresentable requirements for other packages to be
   in sync (e.g. grub2 + grub-signed)
 - have autopkgtest regressions that should block the package being promoted
   to -updates

then yes, -proposed reflects the "likely" soon-to-be state via SRUs.

And since there's no automated way to filter on the above via apt, a
dist-upgrade to -proposed is a dice roll for getting a working set of
packages, or whether reporting any issues you find this way are useful
input to the SRU process.

> I think I have a good use-case for this: System76 would like to be much
> more aggressive about testing -proposed to prevent regressions from
> reaching our customers.  In this case, it's not a matter of testing a
> specific package in -proposed, it's a matter of testing *all* packages in
> -proposed (and their interactions)...  for the purpose of testing the
> expected state of a stable release in the near future.

I understand the intent.  Testing -proposed as a whole then still leaves you
the problem of root-causing any regressions.  Have you had success in
feeding back the results of your testing into the SRU process?

I don't see any reason that you fundamentally /shouldn't/ do this testing, I
just wonder how much it's benefitting you to do so.

> I think the concern that Robie Basak originally brought up (please correct
> me if I'm wrong, Robbie) is that -proposed so often is littered with
> packaging that will most likely *never* make their way into -updates for a
> stable release. Yes, this tends to be worse during the dev cycle for a new
> stable release and immediately after its release, but it can also be quite
> bad during the lifetime of an LTS release.

AIUI Robie's concern was that -proposed is a mess immediately after a stable
release.  And it was *my* argument that this is a quantitative, rather than
qualitative, difference from the rest of the cycle. :)  You agree with the
latter, but argue that this means we should be trying to fix the problem
more broadly; my position is that this is fundamental to the function of
-proposed, and we can't significantly improve it on this axis without
significant tradeoffs elsewhere that we can't justify paying.

> For example, if you look at:

> http://people.canonical.com/~ubuntu-archive/pending-sru.html

> Xenial already has a package in -proposed that has been sitting there for
> 138 days (librarian-puppet-simple). Grated, this package isn't installed by
> default and so understandably is less critical in the grand scheme of
> things. But previous experience watching this SRU page suggests to me that
> this proposed `librarian-puppet-simple` package will *never* make it to
> xenial-updates... so why is this package still there?

Because removing failed-SRU cruft from -proposed is a low-priority task,
compared to all the other things the SRU team works on.

Note that this page does also have a section about '-proposed cleanup',
which does include packages that should be removed due to non-verification. 
It seems a bug that we don't include in this section old packages that have
*failed* verification.

But anyway, removing these packages from -proposed more quickly doesn't help
the user who has already upgraded to them for testing.  There's definitely
no committment that a bad package in -proposed will later be replaced by a
good package in -proposed that you can update to.  

> I guess in summary, I see two important use cases for -proposed:

> 1) For stakeholders to test a single specific (source) package in -proposed,
> especially for SRU verification when said stake holder either filed a
> specific bug or is effected by a specific bug

> 2) For stakeholders to test, in total across all -proposed packages, the
> soon-to-be expected state of the stable release, without awareness of the
> details of any of these -proposed packages (especially without needing to
> guess which specific ones are test-worthy)

> I think making -proposed work like -backports can totally fix use case (1),
> but I don't see how it really helps use case (2).

True; but I think currently we have a situation of serving two masters
poorly, and it would be an overall improvement to handle the first case
better out of the box (because the stakeholders in that case have a more
casual relationship with the SRU process as a whole), while documenting how
to override one's apt config manually for the second case.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek at ubuntu.com                                     vorlon at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-release/attachments/20161019/e5b5513b/attachment.pgp>