[SRU] OpenStack Stable Snapshots

[Corey Bryant]

On Tue, Mar 9, 2021 at 2:55 PM Brian Murray <brian at canonical.com> wrote:

> On Mon, Mar 08, 2021 at 11:58:18AM -0500, Corey Bryant wrote:
> > On Mon, Feb 22, 2021 at 5:16 PM Corey Bryant <corey.bryant at canonical.com
> >
> > wrote:
> >
> > > Hello SRU Team,
> > >
> > > I'm writing to discuss delivery of patches from extended maintenance
> > > stable branches as snapshots.
> > >
> > > This would occur during the extended maintenance phase of an upstream
> > > stable branch, which falls between the maintained phase and EOL phase.
> > > I would see this process being similar to the current process that is
> used
> > > to deliver stable point releases, described at:
> > > https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates.
> > >
> > > Snapshots refers to the process by which we currently deliver package
> > > updates during a development release and prior to the final release.
> For
> > > example, nova 3:22.0.1+git2021012713.d92c0740c6-0ubuntu1 in hirsute
> > > contains everything in the upstream master git branch up to commit hash
> > > d92c0740c6.
> > >
> > > == What is Extended Maintenance? ==
> > > First, it's worth understanding what the upstream "Maintained" phase
> is.
> > > During this phase, upstream regularly releases stable point releases
> that
> > > are based on a specific commit (e.g. the latest HEAD commit for the
> > > stable/queens branch at a point in time), and we then package those
> point
> > > releases up in distro.
> > >
> > > Following the "Maintained" phases is the "Extended Maintenance" phase.
> > > During this phase, upstream no longer cuts any stable point releases
> for
> > > that stable branch, however patches continue to be backported to the
> > > branch. This is the phase where we would like to have the ability to
> > > deliver snapshots of stable branches. This would allow us to pick up
> all
> > > stable branch patches since, say, the final point release for
> stable/queens
> > > (bionic). For example, the current version of nova in bionic is
> > > 2:17.0.13-0ubuntu1, and that is based on the final point release for
> > > upstream stable/queens. If we were to do a snapshot today for nova in
> > > bionic, it would be versioned along the lines of
> > > 2:17.0.13+git2021022200.944443a7b0-0ubuntu1.
> > >
> > > More details on upstream maintenance phases can be reviewed at:
> > >
> https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases
> > >
> > > == Advantages of Stable Snapshots ==
> > > * Less overhead for the OpenStack engineering and SRU teams. Currently
> we
> > > have various cherry-picked patches that are dealt with via separate SRU
> > > bugs requiring individual testing.
> > > * As the cherry-pick process is reactive, many of these SRUs tend to be
> > > high impact or critical for users, requiring immediate action from
> these
> > > teams.
> > > * We have a lot of users that are now on, and moving to, bionic
> (queens)
> > > and we would like to be more proactive in fixing bugs for them.
> > > * This would naturally result in a regular cadence along the lines of
> the
> > > monthly stable point release cadence that we currently follow.
> > > * Cherry-picking individual patches can increase regression potential
> for
> > > frequently modified code when they don't apply cleanly.
> > > * All patches that have landed in an extended maintenance branch have
> > > already landed in all ensuing stable branches and master, therefore
> > > negating much of the regression potential mentioned in "Disadvantages"
> > > below.
> > >
> > > == Disadvantages ==
> > > * Upstream stable branches that are in extended maintenance do not
> > > guarantee the same upstream test coverage that supported branches do.
> The
> > > maintenance-phases documentation states "There is no statement about
> the
> > > level of testing and upgrades from Extended Maintenance are not
> supported
> > > within the Community."
> > >
> > > == What Ubuntu Releases would this affect? ==
> > > Stable snapshots would only be needed for Ubuntu LTS releases (Upstream
> > > "Maintained" period is 18 months for stable branches). Currently this
> would
> > > only be applicable to Ubuntu 18.04 as bionic is the only LTS with a
> > > corresponding upstream stable branch that's in extended maintenance.
> > >
> > > == How many patches are we talking about? ==
> > > For a sampling of stable/queens, new snapshots would pick up this many
> > > patches:
> > >
> > > nova: 337
> > > neutron: 106
> > > heat: 47
> > > cinder: 21
> > > glance: 11
> > > designate: 11
> > > swift: 10
> > > horizon: 8
> > > keystone: 7
> > >
> > > A few of those look daunting, yes, but perhaps that depends on how you
> > > look at it. Also please keep in mind, queens has been in extended
> > > maintenance since October of 2019 so this would be quite the one-time
> > > waterfall for bionic. In the future we'd pick up fewer patches at a
> time at
> > > a regular cadence.
> > >
> > > Thanks for taking the time to read this. Please let me know if you have
> > > any opinions or questions about this approach.
> > >
> > > Corey
> > >
> >
> > Hello SRU Team,
> >
> > I wanted to see if I could revive this topic and discuss the possibility
> of
> > moving forward with stable snapshots as described in this thread.
> >
> > Would it make sense to add this to a meeting agenda or is the mailing
> list
> > a good place to discuss?
>
> The Ubuntu SRU team does have a monthly meeting (conducted via Google
> Meet or whatever it is called now) and the next one is scheduled for
> Thursday, March 18th. Unfortunately I won't be able to attend that
> meeting but perhaps the other SRU team members can invite you so it can
> be discussed.
>
> Cheers,
> --
> Brian Murray
>

Last week I attended the SRU team meeting and also met with Edward
Hope-Morley from Canonical Sustaining Engineering to discuss stable
snapshots. Ed's team is one of the primary drivers/stakeholders for Ubuntu
OpenStack SRU work. Thank you to all who provided input at the meetings.

The main result of the discussion with the SRU team is that we would need
to ensure we are closing the gap, specifically with extended maintenance
branches, where upstream may have a reduced level of testing (including
upgrade testing). Our QA process at
https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates would need to be
enhanced with testing efforts that would mitigate the regression potential
introduced by reduced upstream gate testing.

In discussion with Ed, we came to the conclusion that a proof of concept
makes sense before moving forward. For example we can snapshot the upstream
stable/queens branches (see section labeled "How many patches are we
talking about?" above in thread) and both Ed's team and ours can test them
in a PPAs. This will be an opportunity to assess the code quality and
enhance our QA tests before moving forward with anything official. So
that's the plan for now, and I will plan on reporting back on this thread
once we are ready (or not) to move on from the POC.

Thanks,
Corey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-release/attachments/20210324/3b24a98b/attachment.html>