Choice of the openssl version for 23.10 and 24.04

[Adrien Nader]

NB: re-sending with an e-mail adderss that isn't moderated; moderator:
feel free to reject duplicate mails in the moderation queue

(stripping the quotes a bit)

On Mon, Dec 04, 2023, Dimitri John Ledkov wrote:
> On Mon, 4 Dec 2023 at 09:28, Adrien Nader <adrien at notk.org> wrote:
> > The issue is that we do not know when will be the next openssl LTS. We
> That's irrelevant. Once Ubuntu release goes stable, we no longer apply
> full upstream point releases, and create our own stream of security
> and stable fixes anyway.
> Our timelines of support are also longer than either shorter or longer
> upstream support timelines.

The point is that security updates require work and using a version that
no other distribution uses (which will always be the case unless other
distribution release dates align with Ubuntu's) prevents sharing the
work with other distributions. This was stated earlier (months ago) in
this thread. I don't know how that played out in hindsight. That would
be an interesting thing to know. 

Even if we're not on the same point version as other distributions,
there is still a lot of common things, especially for typical security
patches, and lots of testing in common. I don't think the patches are
very difficult to port across versions (except the ones which porting is
horribly painful) but QA and verification are another matter. Keep in
mind that 3.0 was a big release with many architectural changes and this
has continued in 3.1 and 3.2. There are certainly fewer and fewer
changes but I don't think I would call these changes uncommon yet.

Finally, Ubuntu is not the only distribution with longer support than
openssl's five years for LTS: there can still be cross-distro
cooperation after these years, and it will actually be even more
valuable.

> > can safely bet there will be one before 26.04 and update to the most
> > recent version for 24.10 (since by 26.04, the current LTS won't be
> > supported anymore). However we can't really bet there will be one by
> > 24.04 and even if there were, there probably wouldn't be a new one in
> > time by 26.04.
> >
> > What you are asking for is equivalent to not using openssl LTS releases
> > for Ubuntu LTS releases.
> 
> Yes.
> 
> > There are many more non-LTS releases so we're
> > more likely to end up with a non-LTS version.
> >
> > Openssl time-based releases are very very recent.
> 
> We did ship the first KDE time-based release when it came out for the
> first time.

KDE, especially in Ubuntu, is far less risky than openssl. It is also
updated far more easily.

I've been trying to make an SRU for openssl for several months now and
it hasn't landed yet. I'm not blaming anyone here: there are tons of
reasons for that (including the fact that I don't have the bandwidth to
re-spin something at the moment). 

The fact is that today we're not able to update openssl for anything
except security updates. In other words, no matter what the openssl
maintainer puts in a release, that maintainer won't have anything to do
with it after the release: only the security team will. That makes me
completely uncomfortable with using a release without some kind of
agreement with the security team. Using the most recent version would
make my life easier but I don't think it's the right trade-off here.

By the way, I don't know how that would play with FIPS: I'm not sure it
would be manageable.

> > This was announced
> > late August and only one version was released under this model so far.
> > It wasn't even on time. The delay was fairly small, especially for a
> > first version using this model (and a change mid-cycle). I think they
> > need a bit more time. Will they be late again? Will they continue this
> > scheme? What else? I don't expect openssl upstream can answer these;
> > they don't have enough hindsight yet.
> >
> > We talked about creating a new "openssl" package that is whatever the
> > most recent version is (in universe, and probably with no ESM-guarantee
> > attached somehow). This might need a bit of fiddling with packaging
> > though and in any case, I've had absolutely no time to do that so far.
> >
> > Would such a package fit the needs you see?
> 
> Only one version, the latest openssl, in every Ubuntu release, going forward.
> 
> Shipping two openssl in the archive is extremely difficult and ends up
> being unusable. Devel packages previously have not been coinstallable
> and even if they are it is impossible to get a consistent build
> environment that can build either or both openssl versions, leading to
> extremely bad user experience. For example inability to compile
> scripting language native extensions as part of container builds and
> the like - prime past example
> https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/1794589

Thanks for the link. I was expecting some troubles around that approach
but didn't know this had already happened.

> > Otherwise, really, I think
> > your question is equivalent to not using openssl LTS releases which
> > would be a big departure from the current position.
> 
> It is not a departure - as we have done that before. It will be an
> overall performance saving by reducing the number of HWE SRUs that
> would end up doing anyway. At the cost of increased security
> maintenance - but even that is not too obvious, as often security
> fixes affect only a subset of upstream series.

Which HWE SRUs? Do you think about the use of CPU features? To be
honest, that's not my main concern for performance with openssl 3.0
because at least Intel seems helpful here (stats with n=1 admittedly)
and the scope of patches is very restricted. My main concern is
architectural or algorithmic bottlenecks.

> We did ship bionic with openssl 1.1.0 for example.

Indeed, 1.1.0 wasn't an LTS. That's precedent but at least it was widely
used I think and that enabled cross-distro collaboration for support.
The story would be different for a release every six months without
coordination in advance.

That distros agree on e.g. 3.3 being a fairly common version could make
sense. One such version every two years. That would be probably twice as
frequent as openssl's LTS releases, and four times less frequent than
openssl non-LTS releases. I don't know if that idea would fly.

> My biggest worry is that by mid 2025, when production workloads
> migrate to Noble by default, and if Noble ships with Openssl 3.0 it
> will be a 4 year old release of OpenSSL which is unacceptable.
> 
> Irrespective of upstream longterm status, we must upgrade OpenSSL to a
> new upstream release between two Ubuntu LTS releases.

I concur with you here: I believe that using 3.0 in Noble is
problematic. However I think we do not have a good solution at the
moment. All solutions have serious drawbacks.

For the record, I started this thread when I was about to update openssl
to 3.1 in whichever release was under development back then. And at some
point I realized things were not going to work out well. One thing
that's very difficult is that when Ubuntu non-LTS gets a new openssl
version, we have no idea on which openssl version we'll end up for our
next LTS.

This can improve with openssl having a release schedule but
what we really need is the LTS schedule. This is really a recent change
for openssl and I haven't had time to interact with upstream since 3.2
was released (before that I was monitoring how that release was going).
Strictly speaking, there could be openssl LTS releases every 4.5 years
and that would be really painful. Less than every 2 years seems
doubtful. I would expect something between 2 and 4 years: 2, 2.5, 3,
3.5, 4; that's a lot of possibilities.

(I'll draft something on pen and paper and see if a two-years and
three-years cadence for LTS would be sensible based on the 5=4+1 years
of openssl LTS schedule)

We should first engage with upstream on this. I've had absolutely no
time since mid-October but I hope I can make a bit of room for that
soon. At least we should check if this has been brought up recently.

-- 
Adrien