<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Than you Corey this sounds like a good path forward to me. As you
allude to we have seen a significant increase in the number of
SRUs required for some releases of Openstack, particularly those
that correspond with LTS releases of Ubuntu e.g. Queens and this
is in part due to the significant delta between the last upstream
point release and current stable branch HEAD for extended
maintenance releases of Openstack. This has itself led to
challenges with some SRUs particularly where a fix constitutes
more than one backported patch which themselves potentially have
other dependencies. This is most acute for releases like Nova and
Neutron where the delta can be in the 100s. As you point out,
providing release snapshots is not risk-free but if we provide a
sufficient level of testing and focus resources on testing
snapshots rather than continuous SRUs for the same release I'm
sure we will see a net gain both in terms of reducing load on the
SRU team as well getting fixes out to more users faster. As a
starting point I would really like to see us start building
snapshots e.g. every week or two and making them available for
testing. Once we have enough data to prove their stability we can
then consider their suitability for release. And again as you
point out, the inital snapshot will be the largest delta and
therefore require significant testing but subsequent releases will
be small so lower risk (we would apply the same level of testing
though).</p>
<p>Thanks,<br>
</p>
<p>Ed<br>
</p>
<div class="moz-cite-prefix">On 24/03/2021 13:38, Corey Bryant
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CADn0iZ0faXL0Mu616uWJ5yEqPvGNQ02VLLfNQ+0tBiEQ7Yw43Q@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Mar 9, 2021 at 2:55
PM Brian Murray <<a href="mailto:brian@canonical.com"
target="_blank" moz-do-not-send="true">brian@canonical.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">On Mon, Mar 08, 2021 at
11:58:18AM -0500, Corey Bryant wrote:<br>
> On Mon, Feb 22, 2021 at 5:16 PM Corey Bryant <<a
href="mailto:corey.bryant@canonical.com" target="_blank"
moz-do-not-send="true">corey.bryant@canonical.com</a>><br>
> wrote:<br>
> <br>
> > Hello SRU Team,<br>
> ><br>
> > I'm writing to discuss delivery of patches from
extended maintenance<br>
> > stable branches as snapshots.<br>
> ><br>
> > This would occur during the extended maintenance
phase of an upstream<br>
> > stable branch, which falls between the maintained
phase and EOL phase.<br>
> > I would see this process being similar to the
current process that is used<br>
> > to deliver stable point releases, described at:<br>
> > <a
href="https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates</a>.<br>
> ><br>
> > Snapshots refers to the process by which we
currently deliver package<br>
> > updates during a development release and prior to
the final release. For<br>
> > example, nova
3:22.0.1+git2021012713.d92c0740c6-0ubuntu1 in hirsute<br>
> > contains everything in the upstream master git
branch up to commit hash<br>
> > d92c0740c6.<br>
> ><br>
> > == What is Extended Maintenance? ==<br>
> > First, it's worth understanding what the upstream
"Maintained" phase is.<br>
> > During this phase, upstream regularly releases
stable point releases that<br>
> > are based on a specific commit (e.g. the latest
HEAD commit for the<br>
> > stable/queens branch at a point in time), and we
then package those point<br>
> > releases up in distro.<br>
> ><br>
> > Following the "Maintained" phases is the "Extended
Maintenance" phase.<br>
> > During this phase, upstream no longer cuts any
stable point releases for<br>
> > that stable branch, however patches continue to be
backported to the<br>
> > branch. This is the phase where we would like to
have the ability to<br>
> > deliver snapshots of stable branches. This would
allow us to pick up all<br>
> > stable branch patches since, say, the final point
release for stable/queens<br>
> > (bionic). For example, the current version of nova
in bionic is<br>
> > 2:17.0.13-0ubuntu1, and that is based on the final
point release for<br>
> > upstream stable/queens. If we were to do a
snapshot today for nova in<br>
> > bionic, it would be versioned along the lines of<br>
> > 2:17.0.13+git2021022200.944443a7b0-0ubuntu1.<br>
> ><br>
> > More details on upstream maintenance phases can be
reviewed at:<br>
> > <a
href="https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases</a><br>
> ><br>
> > == Advantages of Stable Snapshots ==<br>
> > * Less overhead for the OpenStack engineering and
SRU teams. Currently we<br>
> > have various cherry-picked patches that are dealt
with via separate SRU<br>
> > bugs requiring individual testing.<br>
> > * As the cherry-pick process is reactive, many of
these SRUs tend to be<br>
> > high impact or critical for users, requiring
immediate action from these<br>
> > teams.<br>
> > * We have a lot of users that are now on, and
moving to, bionic (queens)<br>
> > and we would like to be more proactive in fixing
bugs for them.<br>
> > * This would naturally result in a regular cadence
along the lines of the<br>
> > monthly stable point release cadence that we
currently follow.<br>
> > * Cherry-picking individual patches can increase
regression potential for<br>
> > frequently modified code when they don't apply
cleanly.<br>
> > * All patches that have landed in an extended
maintenance branch have<br>
> > already landed in all ensuing stable branches and
master, therefore<br>
> > negating much of the regression potential
mentioned in "Disadvantages"<br>
> > below.<br>
> ><br>
> > == Disadvantages ==<br>
> > * Upstream stable branches that are in extended
maintenance do not<br>
> > guarantee the same upstream test coverage that
supported branches do. The<br>
> > maintenance-phases documentation states "There is
no statement about the<br>
> > level of testing and upgrades from Extended
Maintenance are not supported<br>
> > within the Community."<br>
> ><br>
> > == What Ubuntu Releases would this affect? ==<br>
> > Stable snapshots would only be needed for Ubuntu
LTS releases (Upstream<br>
> > "Maintained" period is 18 months for stable
branches). Currently this would<br>
> > only be applicable to Ubuntu 18.04 as bionic is
the only LTS with a<br>
> > corresponding upstream stable branch that's in
extended maintenance.<br>
> ><br>
> > == How many patches are we talking about? ==<br>
> > For a sampling of stable/queens, new snapshots
would pick up this many<br>
> > patches:<br>
> ><br>
> > nova: 337<br>
> > neutron: 106<br>
> > heat: 47<br>
> > cinder: 21<br>
> > glance: 11<br>
> > designate: 11<br>
> > swift: 10<br>
> > horizon: 8<br>
> > keystone: 7<br>
> ><br>
> > A few of those look daunting, yes, but perhaps
that depends on how you<br>
> > look at it. Also please keep in mind, queens has
been in extended<br>
> > maintenance since October of 2019 so this would be
quite the one-time<br>
> > waterfall for bionic. In the future we'd pick up
fewer patches at a time at<br>
> > a regular cadence.<br>
> ><br>
> > Thanks for taking the time to read this. Please
let me know if you have<br>
> > any opinions or questions about this approach.<br>
> ><br>
> > Corey<br>
> ><br>
> <br>
> Hello SRU Team,<br>
> <br>
> I wanted to see if I could revive this topic and
discuss the possibility of<br>
> moving forward with stable snapshots as described in
this thread.<br>
> <br>
> Would it make sense to add this to a meeting agenda or
is the mailing list<br>
> a good place to discuss?<br>
<br>
The Ubuntu SRU team does have a monthly meeting (conducted
via Google<br>
Meet or whatever it is called now) and the next one is
scheduled for<br>
Thursday, March 18th. Unfortunately I won't be able to
attend that<br>
meeting but perhaps the other SRU team members can invite
you so it can<br>
be discussed.<br>
<br>
Cheers,<br>
--<br>
Brian Murray<br>
</blockquote>
<div><br>
</div>
<div>Last week I attended the SRU team meeting and also met
with Edward Hope-Morley from Canonical Sustaining
Engineering to discuss stable snapshots. Ed's team is one of
the primary drivers/stakeholders for Ubuntu OpenStack SRU
work. Thank you to all who provided input at the meetings.<br>
</div>
<div><br>
</div>
<div>The main result of the discussion with the SRU team is
that we would need to ensure we are closing the gap,
specifically with extended maintenance branches, where
upstream may have a reduced level of testing (including
upgrade testing). Our QA process at <a
href="https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates"
moz-do-not-send="true">https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates</a>
would need to be enhanced with testing efforts that would
mitigate the regression potential introduced by reduced
upstream gate testing.<br>
</div>
<div><br>
</div>
<div>In discussion with Ed, we came to the conclusion that a
proof of concept makes sense before moving forward. For
example we can snapshot the upstream stable/queens branches
(see section labeled "How many patches are we talking
about?" above in thread) and both Ed's team and ours can
test them in a PPAs. This will be an opportunity to assess
the code quality and enhance our QA tests before moving
forward with anything official. So that's the plan for now,
and I will plan on reporting back on this thread once we are
ready (or not) to move on from the POC.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Corey<br>
</div>
</div>
<div class="gmail_quote"><br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
</blockquote>
</body>
</html>