RFC on Jammy SRU with OpenSSL 3.0 backports for performance

[Adrien Nader]

Hi,

Finally back after pretty long vacation. :) 

On Mon, Aug 21, 2023, Mauricio Faria de Oliveira wrote:
> Hey Simon,
> 
> On Thu, Aug 17, 2023 at 4:42 AM Simon Chopin <simon.chopin at canonical.com> wrote:
> >
> > Hi,
> >
> > Quoting Mauricio Faria de Oliveira (2023-08-15 15:20:50)
> > > Hey,
> > >
> > > I'd like to request input (initially thinking of involved teams: SRU,
> > > Foundations)
> > > on backports of some performance improvement patches to OpenSSL in Jammy.
> > > (Please feel free to comment and include others as appropriate.)
> >
> > FYI, Adrien is currently offline, he should be back in a couple of
> > weeks. Meanwhile you can get my 2¢ as his backup on openssl-related
> > matters :)
> >
> > > SEG has a customer support ticket about it, which allows us to put in the work,
> > > but of course, it is OpenSSL, an SRU, thus agreement beforehand is needed.
> > >
> > > ...
> > >
> > > The context is OpenSSL 3.0 has known, significant performance regressions [0]
> > > from OpenSSL 1.1.1, which has been addressed / still in-progress upstream:
> > > 1) some patches in the 3.0 stable branch
> > > 2) some patches in the master branch (ie, not backported to 3.0)
> > > 3) some issues still open
> > >
> > > To offset regression risk, there are benefits; e.g., 1) Performance:
> > > some improvement 2) Security: smaller delta to 3.0 branch (may help
> > > with CVE fix backports) 3) Community: possibly help with mentions of
> > > Ubuntu 22.04 in regressions
> > >
> > > IMHO, backports should be restricted to the 3.0 branch (so not to
> > > defeat 2).
> >
> > Not Security, but I'm OK with backporting patches from the master branch
> > as well, but only if we can get them in Mantic first, and wait for its
> > GA. That would give exposure to a wider range of hardware, shaking off
> > some regressions.
> 
> Understood. We'll consider that option, for patches from the master branch.
> 
> Hopefully, most patches may be in 3.0 stable (to land before October),
> but nonetheless, it's reassuring to know there's a way forward for
> other patches.

I've been wanting to _ultimately_ update openssl in our stable releases.
That means updating 22.04 with another openssl LTS. Before anyone
panics: that's a really long term goal.

Since openssl had relatively small SRUs in the past years, I had planned
to do things very progressively. At the moment we have 3.0.10 in Mantic
(I didn't check who updated it but thanks!) and 3.0.8 in Lunar. No
regression compared to Jammy's 3.0.2. I didn't read about issues on
other distros either. That's encouraging.

We're talking about two kinds of patches: performance and bug fixes.
Upstream backports bug fixes to the 3.0 release but they don't
necessarily do it for performance fix (they probably typically don't).
There's no "complete" list for such fixes either (there's not even a
definition of which patches would qualify).
While the patches that Mauricio linked to have been backported to 3.0, a
number of the performance fixes have not and it will probably be
difficult to backport them since they can be architectural or locking
changes.
I'm pretty sure the number of performance fixes will only lower and the
situation is transitory but right now there's a demand for several
performance fixes. For instance, it seems Haproxy has encountered
redhibitory performance.

Below are the "steps" I had thought about before my vacation (hopefully
I remember well :) ). Steps 1 and 2 could be merged or shortened
depending on the SRU team.

1- SRU with select patches

There have been a number of issues reported with upstream patches but
they had low impact and/or had easy workarounds. Since there are now
other incentives to do an SRU, I plan to include everything in it.
I think all patches are in Lunar or even previous releases.

I don't have a list for performance patches however, except for a couple
ones. I don't plan to list them by myself; instead I welcome inputs on
that. Benchmarking is simply too much work, especially considering the
kind of performance regressions from 1.1 to 3.0 that I've seen talked
about on the openssl bug tracker. That would probably be a year-long
full-time job.

I would like to start preparing this SRU soon and I hope to have it
ready in October (that's my first SRU this large but I think/hope that's
realistic). Since it would be the first one, I plan to stick to changes
that don't require backporting work. For subsequent SRUs we could
include more patches as Simon said (i.e.  patches from master provided
they go into non-LTS releases first during their development cycle).

2- Micro-release updates

If all goes well, I'd like to do micro-release updates for openssl; that
would obviously include more changes and possibly include some
performance improvements.

Note that upstream really wants us to track micro-releases and has
unambiguously stated they care a lot about stability:
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2019970

3- Full release updates

As I stated, I'd like to be able to update openssl and maybe have the
same openssl LTS version across Ubuntu LTS versions. Obviously that
won't be worked on before the previous steps are successful. The initial
motive was precisely performance regressions but also more generally the
fact that it could be less work and less risk overall (I'm worried about
patching 3.0 on 22.04 in 8 or 10 years, while most probably having to
backport whole new features).

Best,

-- 
Adrien