Please lift aging requirement on gce-compute-image-packages in the stable release exception

Wed Aug 8 15:09:59 UTC 2018

One extra argument for waiving the requirement: this package does most
of its work on first-boot of an instance; we have to manually build new
GCE images specifically to be able to test it.  Users who just enable
-proposed and upgrade (even if they then reboot) will not be performing
meaningful testing of the package.

On Wed, Aug 08, 2018 at 04:10:26PM +0200, Lukasz Zemczak wrote:
> Hello,
> 
> In my humble opinion, for this particular case I would be inclined
> toward lifting the aging requirement. Yes, it's not a completely
> straightforward thing, even though I usually approved of releasing
> early of this package before after validation was performed. From my
> point of view a strong enough rationale has been provided for that.
> Balint's original e-mail states:
> "No one else is expected to test those packages in the remaining time
> of default required minimal age."
> I think in case of this package this is true. As already mentioned by
> Steve, the gce-compute-image-packages binaries are *only* used on
> cloud instances. This basically means almost absolutely no one will be
> using the versions from -proposed, meaning the 7 days are wasted on
> idle waiting. My rationale below:
> 
> To me the aging period is always a safety buffer for when a package
> enters -proposed, during which it can get additional testing (or
> 'dogfooding') by both interested and unrelated parties. In my eyes
> though, usually what this means in practice is to give users that are
> affected by the selected bug or are generally interested in the given
> package time to pick it up from -proposed and give it a spin
> themselves. Formally for us to consider an SRU to be verified requires
> only one tested giving a +1 on the package, after providing test
> results. The aging period (besides the obvious case of letting it run
> on systems of users that always run with -proposed enabled for testing
> purposes) serves the purpose of giving additional time for those
> 'other than the verifying person' affected users to actually test the
> package and report any regressions that the original verifier could
> have missed. For this, there have to be such users - and those users
> need to be able to perform the validation steps as noted in the test
> case.
> 
> In the case of gce-compute-image-packages, considering the nature of
> the package itself, the test case, the SRU exception and the release
> process of the actual package itself, I think we can be pretty
> confident that the only people aware of SRU updates of those packages
> are: the uploader, the SRU team and GCE upstream. These people are
> probably also the only ones that can fully test the packages (since
> those are usually new upstream releases), not counting cases where
> there are specific isolated LP bugs involved in the upload (for those
> I would say aging can still make some sense).
> 
> All those small arguments, in my opinion, sum up into a decent enough
> rationale for ignoring the aging period in this case.
> 
> Cheers,
> 
> 
> On 7 August 2018 at 22:27, Robie Basak <robie.basak at ubuntu.com> wrote:
> > Hi Steve,
> >
> > Summary: I still think we should have an actual justification for
> > waiving the aging requirement. I still don't think that we've actually
> > been given a justification in this case. Give me one and I'll
> > reconsider, but IMHO without any justification, we should either change
> > policy to drop it wholesale, or maintain our current policy of having
> > it.
> >
> > The rest of my response is mostly about why we have the general policy
> > in the first place, rather than this specific case.
> >
> > On Tue, Aug 07, 2018 at 11:41:31AM -0700, Steve Langasek wrote:
> >> That is certainly the purpose of the aging period, but I think this
> >> overstates its value in the SRU process overall.  You say you believe you
> >> have seen this happen in practice, to block a buggy SRU before it reached
> >> -updates.  But broadly speaking:
> >>
> >>  - dist-upgrading (or trying to use update-manager) with -proposed is a bad
> >>    idea for a production machine, because packages in -proposed are known
> >>    not to be consistently installable throughout the SRU lifecycle
> >>  - therefore, running with -proposed enabled is not something that is
> >>    broadly socialized (nor should be) as a way for users to contribute to
> >>    the quality of Ubuntu
> >>  - therefore, random user testing coverage of SRUs is sparse, whether we
> >>    wait 7 days or 30.
> >
> > Nevertheless, I think that wider random user testing coverage of
> > packages in proposed should be something we aim to achieve. This is
> > something I've mentioned before[1]. I'm confident that we have enough of
> > a community willing to live on the edge that they would be willing to do
> > this: if we make it easy for them and publish this as "a way to help
> > out". Regardless of whether or not we can do this today, removing the
> > aging period moves us in the opposite direction.
> >
> >> There is *some* value to an aging period, which is why I have never
> >> advocated dropping the default aging period entirely.  But the value is
> >> small, not straightforwardly quantifiable, and doesn't outweigh the cost in
> >> human developer time of multiple round-trips with the SRU team; so whenever
> >> anyone asks to waive the waiting period, and I don't have /specific/
> >> concerns about the SRU, my answer will be yes.
> >
> > I disagree. Since aging is something we consider worthy enough to have
> > in our policy by default, I think that waiving the aging period in a
> > specific case should require at least _some_ justification. "I want to
> > do it" or "It isn't needed" is, IMHO, not in itself a valid reason.
> > Accepting those statements would in practice mean that aging doesn't
> > exist in the SRU process for all those who disagree with it; in that
> > case, why have an aging period in our policy at all? If we're going to
> > accept that, we might as well remove the requirement entirely.
> >
> > On the other hand, give me an actual justification and I'd be happy to
> > consider it. I just don't feel like we've been given a justification at
> > all in this case, apart from "I don't like it".
> >
> > As for requiring an extra round-trip with the SRU team, I don't think
> > this applies to SRUs in -proposed for which all the bugs are green and
> > the SRU process properly followed in the SRU report by the time the
> > aging period expires. In practice, I see these all being released to
> > -updates promptly and without further interaction.
> >
> > Most delays are caused by Ubuntu developers not following documented SRU
> > procedure, and it is that which causes extra round trips. That can be
> > fixed by uploading Ubuntu developers easily enough; I don't think it's
> > necessary to compromise quality (since we both agree that there is
> > *some* value to an aging period) to do it.
> >
> >> I note that there's no language at all in the wiki page at
> >> https://wiki.ubuntu.com/StableReleaseUpdates which talks about waiving the
> >> waiting period.  I think that is an oversight; I recall seeing written
> >> language at one point about this, but I can't find it now (including in the
> >> history of this wiki page).  Other long-time members of the SRU team can
> >> check me on this, but my understanding of the actual policy is that any
> >> member of the SRU team has the authority to waive the 7-day period any time
> >> they think it is appropriate to do so.  I think we should fix the wiki
> >> documentation to match.
> >
> > My understanding matches yours. I have always understood any single SRU
> > team member to have the delegated authority to waive the aging
> > requirement on a case-by-case basis. What I think we're discussing here
> > is when the SRU team believes they should.
> >
> >> > In this case, I think user-based regression alert opportunity still
> >> > applies. Presumably the reason for this package to exist in the Ubuntu
> >> > archive is that there are users. Removing the aging requirement removes
> >> > the opportunity for those users to tell us about problems.
> >>
> >> The package in question, gce-compute-image-packages, is only installed in
> >> cloud instances.  I believe this means that the organic regression test
> >> coverage by users is LESS than it would be for another package, because the
> >> test coverage we do get tends to be from individually-deployed machines (a
> >> personal desktop/laptop or a one-off server), which is less likely to happen
> >> in a cloud where deployments are intended to be done at scale.[1]
> >
> > Users managing cloud deployments done at scale following best practice
> > already have the ability to test such deployments automatically,
> > including packages in proposed, at virtually no extra maintenance cost.
> >
> > That we currently don't recommend that users enable -proposed wholesale
> > is a use case that, in my view, we should fix (whether by "fixing"
> > -proposed or by providing some other mechanism). I see the current
> > inability as a "bug", and I don't think the existence of this bug should
> > be used as a justification for removing the aging period; that would
> > make it impossible to solve the problem.
> >
> >> And in cases where we already have an SRU exception, which dictates that
> >> there must be strong upstream QA for the package before it is pushed to the
> >> SRU queue, I think it's all the more reasonable to omit the default aging
> >> period as part of that SRU exception.
> >
> > Strong upstream QA certainly does shift the balance of regression risk
> > in the "safer" direction. However, packages with SRU exceptions
> > generally end up with riskier stable uploads, such as feature additions.
> > So that's the counterpoint that could shift the balance back and cancel
> > this out. Going in the other direction, it might well be a reasonable
> > mitigation for the extra regression risk in permitting wholesale
> > upstream feature releases into Ubuntu stable releases to require a
> > _longer_ aging period. I'm not suggesting we do this, but I hope this
> > example helps illustrate my point.
> >
> > Robie
> >
> > [1] https://lists.ubuntu.com/archives/ubuntu-release/2016-October/003948.html
> >
> > --
> > Ubuntu-release mailing list
> > Ubuntu-release at lists.ubuntu.com
> > Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
> >
> 
> 
> 
> -- 
> Łukasz 'sil2100' Zemczak
>  Foundations Team
>  lukasz.zemczak at canonical.com
>  www.canonical.com
> 
> -- 
> Ubuntu-release mailing list
> Ubuntu-release at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-release