[SRU] OpenStack Stable Snapshots
Edward Hope-Morley
edward.hope-morley at canonical.com
Mon Mar 29 11:04:40 UTC 2021
Than you Corey this sounds like a good path forward to me. As you allude
to we have seen a significant increase in the number of SRUs required
for some releases of Openstack, particularly those that correspond with
LTS releases of Ubuntu e.g. Queens and this is in part due to the
significant delta between the last upstream point release and current
stable branch HEAD for extended maintenance releases of Openstack. This
has itself led to challenges with some SRUs particularly where a fix
constitutes more than one backported patch which themselves potentially
have other dependencies. This is most acute for releases like Nova and
Neutron where the delta can be in the 100s. As you point out, providing
release snapshots is not risk-free but if we provide a sufficient level
of testing and focus resources on testing snapshots rather than
continuous SRUs for the same release I'm sure we will see a net gain
both in terms of reducing load on the SRU team as well getting fixes out
to more users faster. As a starting point I would really like to see us
start building snapshots e.g. every week or two and making them
available for testing. Once we have enough data to prove their stability
we can then consider their suitability for release. And again as you
point out, the inital snapshot will be the largest delta and therefore
require significant testing but subsequent releases will be small so
lower risk (we would apply the same level of testing though).
Thanks,
Ed
On 24/03/2021 13:38, Corey Bryant wrote:
>
>
> On Tue, Mar 9, 2021 at 2:55 PM Brian Murray <brian at canonical.com
> <mailto:brian at canonical.com>> wrote:
>
> On Mon, Mar 08, 2021 at 11:58:18AM -0500, Corey Bryant wrote:
> > On Mon, Feb 22, 2021 at 5:16 PM Corey Bryant
> <corey.bryant at canonical.com <mailto:corey.bryant at canonical.com>>
> > wrote:
> >
> > > Hello SRU Team,
> > >
> > > I'm writing to discuss delivery of patches from extended
> maintenance
> > > stable branches as snapshots.
> > >
> > > This would occur during the extended maintenance phase of an
> upstream
> > > stable branch, which falls between the maintained phase and
> EOL phase.
> > > I would see this process being similar to the current process
> that is used
> > > to deliver stable point releases, described at:
> > > https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates.
> > >
> > > Snapshots refers to the process by which we currently deliver
> package
> > > updates during a development release and prior to the final
> release. For
> > > example, nova 3:22.0.1+git2021012713.d92c0740c6-0ubuntu1 in
> hirsute
> > > contains everything in the upstream master git branch up to
> commit hash
> > > d92c0740c6.
> > >
> > > == What is Extended Maintenance? ==
> > > First, it's worth understanding what the upstream "Maintained"
> phase is.
> > > During this phase, upstream regularly releases stable point
> releases that
> > > are based on a specific commit (e.g. the latest HEAD commit
> for the
> > > stable/queens branch at a point in time), and we then package
> those point
> > > releases up in distro.
> > >
> > > Following the "Maintained" phases is the "Extended
> Maintenance" phase.
> > > During this phase, upstream no longer cuts any stable point
> releases for
> > > that stable branch, however patches continue to be backported
> to the
> > > branch. This is the phase where we would like to have the
> ability to
> > > deliver snapshots of stable branches. This would allow us to
> pick up all
> > > stable branch patches since, say, the final point release for
> stable/queens
> > > (bionic). For example, the current version of nova in bionic is
> > > 2:17.0.13-0ubuntu1, and that is based on the final point
> release for
> > > upstream stable/queens. If we were to do a snapshot today for
> nova in
> > > bionic, it would be versioned along the lines of
> > > 2:17.0.13+git2021022200.944443a7b0-0ubuntu1.
> > >
> > > More details on upstream maintenance phases can be reviewed at:
> > >
> https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases
> > >
> > > == Advantages of Stable Snapshots ==
> > > * Less overhead for the OpenStack engineering and SRU teams.
> Currently we
> > > have various cherry-picked patches that are dealt with via
> separate SRU
> > > bugs requiring individual testing.
> > > * As the cherry-pick process is reactive, many of these SRUs
> tend to be
> > > high impact or critical for users, requiring immediate action
> from these
> > > teams.
> > > * We have a lot of users that are now on, and moving to,
> bionic (queens)
> > > and we would like to be more proactive in fixing bugs for them.
> > > * This would naturally result in a regular cadence along the
> lines of the
> > > monthly stable point release cadence that we currently follow.
> > > * Cherry-picking individual patches can increase regression
> potential for
> > > frequently modified code when they don't apply cleanly.
> > > * All patches that have landed in an extended maintenance
> branch have
> > > already landed in all ensuing stable branches and master,
> therefore
> > > negating much of the regression potential mentioned in
> "Disadvantages"
> > > below.
> > >
> > > == Disadvantages ==
> > > * Upstream stable branches that are in extended maintenance do not
> > > guarantee the same upstream test coverage that supported
> branches do. The
> > > maintenance-phases documentation states "There is no statement
> about the
> > > level of testing and upgrades from Extended Maintenance are
> not supported
> > > within the Community."
> > >
> > > == What Ubuntu Releases would this affect? ==
> > > Stable snapshots would only be needed for Ubuntu LTS releases
> (Upstream
> > > "Maintained" period is 18 months for stable branches).
> Currently this would
> > > only be applicable to Ubuntu 18.04 as bionic is the only LTS
> with a
> > > corresponding upstream stable branch that's in extended
> maintenance.
> > >
> > > == How many patches are we talking about? ==
> > > For a sampling of stable/queens, new snapshots would pick up
> this many
> > > patches:
> > >
> > > nova: 337
> > > neutron: 106
> > > heat: 47
> > > cinder: 21
> > > glance: 11
> > > designate: 11
> > > swift: 10
> > > horizon: 8
> > > keystone: 7
> > >
> > > A few of those look daunting, yes, but perhaps that depends on
> how you
> > > look at it. Also please keep in mind, queens has been in extended
> > > maintenance since October of 2019 so this would be quite the
> one-time
> > > waterfall for bionic. In the future we'd pick up fewer patches
> at a time at
> > > a regular cadence.
> > >
> > > Thanks for taking the time to read this. Please let me know if
> you have
> > > any opinions or questions about this approach.
> > >
> > > Corey
> > >
> >
> > Hello SRU Team,
> >
> > I wanted to see if I could revive this topic and discuss the
> possibility of
> > moving forward with stable snapshots as described in this thread.
> >
> > Would it make sense to add this to a meeting agenda or is the
> mailing list
> > a good place to discuss?
>
> The Ubuntu SRU team does have a monthly meeting (conducted via Google
> Meet or whatever it is called now) and the next one is scheduled for
> Thursday, March 18th. Unfortunately I won't be able to attend that
> meeting but perhaps the other SRU team members can invite you so
> it can
> be discussed.
>
> Cheers,
> --
> Brian Murray
>
>
> Last week I attended the SRU team meeting and also met with Edward
> Hope-Morley from Canonical Sustaining Engineering to discuss stable
> snapshots. Ed's team is one of the primary drivers/stakeholders for
> Ubuntu OpenStack SRU work. Thank you to all who provided input at the
> meetings.
>
> The main result of the discussion with the SRU team is that we would
> need to ensure we are closing the gap, specifically with extended
> maintenance branches, where upstream may have a reduced level of
> testing (including upgrade testing). Our QA process at
> https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates would need to
> be enhanced with testing efforts that would mitigate the regression
> potential introduced by reduced upstream gate testing.
>
> In discussion with Ed, we came to the conclusion that a proof of
> concept makes sense before moving forward. For example we can snapshot
> the upstream stable/queens branches (see section labeled "How many
> patches are we talking about?" above in thread) and both Ed's team and
> ours can test them in a PPAs. This will be an opportunity to assess
> the code quality and enhance our QA tests before moving forward with
> anything official. So that's the plan for now, and I will plan on
> reporting back on this thread once we are ready (or not) to move on
> from the POC.
>
> Thanks,
> Corey
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-release/attachments/20210329/c7420059/attachment.html>
More information about the Ubuntu-release
mailing list