Choice of the openssl version for 23.10 and 24.04

[Adrien Nader]

On Mon, Dec 04, 2023, Dimitri John Ledkov wrote:
> On Mon, 4 Dec 2023 at 12:34, Adrien Nader <adrien at notk.org> wrote:
> >
> > (stripping the quotes a bit)
> >
> > On Mon, Dec 04, 2023, Dimitri John Ledkov wrote:
> > > On Mon, 4 Dec 2023 at 09:28, Adrien Nader <adrien at notk.org> wrote:
> > > > The issue is that we do not know when will be the next openssl LTS. We
> > > That's irrelevant. Once Ubuntu release goes stable, we no longer apply
> > > full upstream point releases, and create our own stream of security
> > > and stable fixes anyway.
> > > Our timelines of support are also longer than either shorter or longer
> > > upstream support timelines.
> >
> > The point is that security updates require work and using a version that
> > no other distribution uses (which will always be the case unless other
> > distribution release dates align with Ubuntu's) prevents sharing the
> > work with other distributions. This was stated earlier (months ago) in
> > this thread. I don't know how that played out in hindsight. That would
> > be an interesting thing to know.
> >
> > Even if we're not on the same point version as other distributions,
> > there is still a lot of common things, especially for typical security
> > patches, and lots of testing in common. I don't think the patches are
> > very difficult to port across versions (except the ones which porting is
> > horribly painful) but QA and verification are another matter. Keep in
> > mind that 3.0 was a big release with many architectural changes and this
> > has continued in 3.1 and 3.2. There are certainly fewer and fewer
> > changes but I don't think I would call these changes uncommon yet.
> >
> > Finally, Ubuntu is not the only distribution with longer support than
> > openssl's five years for LTS: there can still be cross-distro
> > cooperation after these years, and it will actually be even more
> > valuable.
> >
> > > > can safely bet there will be one before 26.04 and update to the most
> > > > recent version for 24.10 (since by 26.04, the current LTS won't be
> > > > supported anymore). However we can't really bet there will be one by
> > > > 24.04 and even if there were, there probably wouldn't be a new one in
> > > > time by 26.04.
> > > >
> > > > What you are asking for is equivalent to not using openssl LTS releases
> > > > for Ubuntu LTS releases.
> > >
> > > Yes.
> > >
> > > > There are many more non-LTS releases so we're
> > > > more likely to end up with a non-LTS version.
> > > >
> > > > Openssl time-based releases are very very recent.
> > >
> > > We did ship the first KDE time-based release when it came out for the
> > > first time.
> >
> > KDE, especially in Ubuntu, is far less risky than openssl. It is also
> > updated far more easily.
> >
> > I've been trying to make an SRU for openssl for several months now and
> > it hasn't landed yet. I'm not blaming anyone here: there are tons of
> > reasons for that (including the fact that I don't have the bandwidth to
> > re-spin something at the moment).
> >
> > The fact is that today we're not able to update openssl for anything
> > except security updates. In other words, no matter what the openssl
> > maintainer puts in a release, that maintainer won't have anything to do
> > with it after the release: only the security team will. That makes me
> > completely uncomfortable with using a release without some kind of
> > agreement with the security team. Using the most recent version would
> > make my life easier but I don't think it's the right trade-off here.
> >
> > By the way, I don't know how that would play with FIPS: I'm not sure it
> > would be manageable.
> >
> > > > This was announced
> > > > late August and only one version was released under this model so far.
> > > > It wasn't even on time. The delay was fairly small, especially for a
> > > > first version using this model (and a change mid-cycle). I think they
> > > > need a bit more time. Will they be late again? Will they continue this
> > > > scheme? What else? I don't expect openssl upstream can answer these;
> > > > they don't have enough hindsight yet.
> > > >
> > > > We talked about creating a new "openssl" package that is whatever the
> > > > most recent version is (in universe, and probably with no ESM-guarantee
> > > > attached somehow). This might need a bit of fiddling with packaging
> > > > though and in any case, I've had absolutely no time to do that so far.
> > > >
> > > > Would such a package fit the needs you see?
> > >
> > > Only one version, the latest openssl, in every Ubuntu release, going forward.
> > >
> > > Shipping two openssl in the archive is extremely difficult and ends up
> > > being unusable. Devel packages previously have not been coinstallable
> > > and even if they are it is impossible to get a consistent build
> > > environment that can build either or both openssl versions, leading to
> > > extremely bad user experience. For example inability to compile
> > > scripting language native extensions as part of container builds and
> > > the like - prime past example
> > > https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/1794589
> >
> > Thanks for the link. I was expecting some troubles around that approach
> > but didn't know this had already happened.
> >
> > > > Otherwise, really, I think
> > > > your question is equivalent to not using openssl LTS releases which
> > > > would be a big departure from the current position.
> > >
> > > It is not a departure - as we have done that before. It will be an
> > > overall performance saving by reducing the number of HWE SRUs that
> > > would end up doing anyway. At the cost of increased security
> > > maintenance - but even that is not too obvious, as often security
> > > fixes affect only a subset of upstream series.
> >
> > Which HWE SRUs? Do you think about the use of CPU features? To be
> > honest, that's not my main concern for performance with openssl 3.0
> > because at least Intel seems helpful here (stats with n=1 admittedly)
> > and the scope of patches is very restricted. My main concern is
> > architectural or algorithmic bottlenecks.
> >
> > > We did ship bionic with openssl 1.1.0 for example.
> >
> > Indeed, 1.1.0 wasn't an LTS. That's precedent but at least it was widely
> > used I think and that enabled cross-distro collaboration for support.
> > The story would be different for a release every six months without
> > coordination in advance.
> >
> > That distros agree on e.g. 3.3 being a fairly common version could make
> > sense. One such version every two years. That would be probably twice as
> > frequent as openssl's LTS releases, and four times less frequent than
> > openssl non-LTS releases. I don't know if that idea would fly.
> >
> > > My biggest worry is that by mid 2025, when production workloads
> > > migrate to Noble by default, and if Noble ships with Openssl 3.0 it
> > > will be a 4 year old release of OpenSSL which is unacceptable.
> > >
> > > Irrespective of upstream longterm status, we must upgrade OpenSSL to a
> > > new upstream release between two Ubuntu LTS releases.
> >
> > I concur with you here: I believe that using 3.0 in Noble is
> > problematic. However I think we do not have a good solution at the
> > moment. All solutions have serious drawbacks.
> >
> > For the record, I started this thread when I was about to update openssl
> > to 3.1 in whichever release was under development back then. And at some
> > point I realized things were not going to work out well. One thing
> > that's very difficult is that when Ubuntu non-LTS gets a new openssl
> > version, we have no idea on which openssl version we'll end up for our
> > next LTS.
> >
> > This can improve with openssl having a release schedule but
> > what we really need is the LTS schedule. This is really a recent change
> > for openssl and I haven't had time to interact with upstream since 3.2
> > was released (before that I was monitoring how that release was going).
> > Strictly speaking, there could be openssl LTS releases every 4.5 years
> > and that would be really painful. Less than every 2 years seems
> > doubtful. I would expect something between 2 and 4 years: 2, 2.5, 3,
> > 3.5, 4; that's a lot of possibilities.
> >
> > (I'll draft something on pen and paper and see if a two-years and
> > three-years cadence for LTS would be sensible based on the 5=4+1 years
> > of openssl LTS schedule)
> >
> > We should first engage with upstream on this. I've had absolutely no
> > time since mid-October but I hope I can make a bit of room for that
> > soon. At least we should check if this has been brought up recently.
> 
> https://www.openssl.org/blog/blog/2023/11/23/OpenSSL32/
> """
> The next feature release after OpenSSL 3.2 will be OpenSSL 3.3, which
> will be released no later than 30 April 2024. This release is expected
> to include QUIC server support. The determination of what other
> features will be shipped in OpenSSL 3.3 is ongoing and will be decided
> by our recently announced Release Steering Committee.
> """
> Thus timing wise it seems doable to ship noble with v3.3, unless things slip
> 
> Looking at feature list between v3.0 and v3.2, we absolutely desire:
> 
> * Support for client side QUIC, including support for multiple streams
> (RFC 9000)

That one was at or near the top of openssl changes that made me want to
update but it turns out it's a new API and therefore not compatible with
one of the existing QUIC libraries. As such, openssl might not be where
to look for when it comes to QUIC support. (I don't know how's our
support for QUIC currently)

> * Support for Ed25519ctx, Ed25519ph and Ed448ph in addition to
> existing support for Ed25519 and Ed448 (RFC 8032)
> * Support for deterministic ECDSA signatures (RFC 6979)
> * Support for the Argon2 KDF, along with supporting thread pool
> functionality (RFC 9106)
> * Support for TCP Fast Open on Linux, macOS and FreeBSD, where enabled
> and supported (RFC 7413)
> * Support for TLS certificate compression, including library support
> for zlib, Brotli and zstd (RFC 8879)
> 
> Without these features, most users will seek out to get that, and will
> find ways to get it, without running our build of OpenSSL, exposing
> them to security risks.
> 
> Can you backport all of that to the v3.0 series? how is that going to
> make it look any much different from shipping v3.2 or v3.3?

This has been my concern too: I don't want us to end up with a
franken-openssl. I'm now very confident this won't happen, in particular
for Noble. The only question is therefore how needed are these features.
If you look at the changes in openssl 3.x versions, you'll always see a
long list of change at a pretty rapid pace. I'm sure 3.3 will include
many features we'll want too. I have absolutely no doubt about it.

But what's the point in shipping something we cannot support? Today we
can't really support an non-LTS openssl version. We wouldn't have this
conversation if openssl were in universe but it's not, it's in main.

> https://www.openssl.org/policies/releasestrat.html
> """
> We may designate a release as a Long Term Support (LTS) release. LTS
> releases will be supported for at least five years and we will specify
> one at least every four years. Non-LTS releases will be supported for
> at least two years.
> """

I remembered only one year for non-LTS but like for LTS, that final year
is security-only.

> If synergy and coordination is a serious blocker, and v3.3 is not
> designated as upstream longterm, we should plan for in-noble upgrade
> to a later upstream - however painful that is. Although API and ABI
> stability guarantees have been improved upstream post v3.0 release.
> Maybe it is worth to run abi & api checks against jammy using v3.2.
> Also given the previous release date of v3.0 the upstream OpenSSL
> longterm might not arrive until 2025. Do you really want to support
> v3.0 openssl in noble, with many users opting not to use it at all,
> and suffer support fallout from that?

As I mentioned earlier, I have been trying to SRU a few changes with a
moonshot of being able to update across releases. Do an SRU, get an MRE,
manager to get an RE (MRE without the "micro").

What did I learn?

ABI (and API I guess) was stable between jammy and lunar or mantic.
Pretty good.

However the main teaching is that I could update the SRU wiki page with
tons of things to check for.
For instance, in that SRU there was initially 4 patches series. While
working on SRU'ing one to fix blowfish ECB/OCB/XYZ encryption in jammy,
I noticed that the bug had created a situation where these BF
encryptions and decryptions were incompatible with the rest of the world
but compatible within Jammy, and that fixing it would restore
compatibility with the rest of the world but break it between
pre-upgrade and post-upgrade Jammy.
Worse: this is used both for encrypting and decrypting data at rest but
also data in transit. And someone reported that issue a couple weeks ago
with a slightly different scenario.
No matter what we do, there are incompatibilities. The SRU policy is to
not potentially break workflows.

The issue appeared in 3.0.0, we shipped 3.0.2 in Jammy, and it was fixed
in 3.0.4 (shipped in kinetic).

I would be surprised that is the only incompatible change with 3.0.z,
let alone 3.y.z. At least there aren't more changes in 3.0.z than can be
tracked but even then, there's a lot of things to track. Moreover, 3.2
on Jammy without these breaking changes would also be quite surprising
for users.

As I said: I think it would be interesting to have the most recent
version but we're not able to do the amount of work required. Probably
at least by a factor of 10.

> Because we cannot upgrade OpenSSL only every 4 years, that's way too
> slow for the speed of performance gains and improvements in TLS.

Now that I've finally eaten, I took the time to write down a support
schedule for openssl's upstream. They would have to support _five_
releases at the same time once the pace is set. I'm also not sure who
runs a non-LTS _and_ needs two years or support. That lines up well
enough with a release a year but not really with two releases a year.

I think the most sensible thing to do is approach upstream and question
their cadence and support duration. A release every six months is fine
and I'm under the impression they went for time-based releases after
having too many delays with 3.x releases. But the support duration is
another matter: six months and six months security-only seems a better
fit for their release cadence, with only three releases requiring
support. I hope the time freed by this would make it possible to release
an LTS every two years instead.

> For interim releases, we should ship the latest OpenSSL release and
> ship the upstream point releases fixes, as that effectively tests
> things for the next LTS and the support timeframes overlap there,
> meaning it should become at no additional burden for the security
> team.

The issue remains the same: if we do that, we are likely to ship a
non-LTS in our LTS. This will certainly happen starting with 24.10 since
there will be another openssl LTS by 26.04 but without such a guarantee,
this is equivalent to using a non-LTS in our LTS.

-- 
Adrien