glibc SRU policy

[Simon Chopin]

Hi SRU team,

In our process we have the notion of MRE for upstreams that have new
bugfix releases that we might want to include. I would love to use a
similar mechanism for glibc SRUs, as there's a fair amount of fixes that
upstream backports onto their stable releases over time. However, there
are a few mismatches with the criteria outlined on the wiki, so I'd like
the input of the SRU team before proceeding.

# No actual release.

Upstream does *not* cut new bugfix releases. Instead, they expect
distributors to just use the tip of the relevant stable branch. While
there's CI done on the commits themselves, not having an actual release
necessarily diffuses the testing onto multiple slightly different
versions of the codebase, which isn't ideal.


# autopkgtests against the actual binaries

The glibc autopkgtests themselves are just a full package rebuild, and I
couldn't find any obvious way to make the test suite run against the
installed libc. However, this might be offset by the sheer quantity of
rdep tests that are triggered?


# Upstream CI

While there is some upstream CI, it's not well documented, especially
regarding release branches. After some digging, it seems the CI is (was)
only enabled on the master branch, in addition to some CI on individual
patches via Patchwork.

I've already sent in a patch to the buildbot configuration to enable the
CI on the release branches, which has been merged. Looking at said
configuration, I can say that the CI does simple builds on amd64, i386,
s390x, ppc64le (and others), and runs the full test suite on amd64 and
arm64.

Based on what I could gather, the Patchwork checks are basically limited
to a full build (with testsuite) on i686 for every patch applied on top
of master.


# Upstream backporting policy

Lastly, and that's perhaps more problematic for me, the contents of the
stable branch aren't *just* bugfixes. To quote upstream:

> The release branches will be upsdated with conservative bug fixes and new
> features while retaining backards compatibility.
> [...]
> The last 3 releases have seen ~700 commits backported to fix bugs or
> implement ABI-neutral features (like IFUNCs).

In particular, the features in question will usually be new optimized
implementation of core routines. Given our SRU policy, this could be
considered a form of HWE, I suppose.

However, the new features are backported to the stable branches directly
from the development tree *before* they make it to a fresh release,
which means that there's a decent chance that stable users would be the
first ones to get the new code, rather than those tracking the latest
release. The issue was raised by Michael on the upstream mailing list:

https://sourceware.org/pipermail/libc-alpha/2023-February/145672.html

with the rest of the thread here:

https://sourceware.org/pipermail/libc-alpha/2023-March/146031.html

In more concrete terms, I went through the commits for the 2.35 stable
branch that were not included in the Jammy release, and found 300+
commits split into 110 patchsets. From those 110 patchsets, I have 38
marked as "refused", with 10 of them being fixes for architectures we
don't support, the rest being new features, optimizations, or fixes for
those.

I'd very much like for our users to get the dozens of fixes that are
present on the branch, but it's not possible to create proper SRU
templates for each of them, as there's often not enough context for me
to get a reproducer. Given the nature of the package, it's possible that
some of our users have experienced the bugs but haven't been able to
attribute the cause to glibc.

Now, besides using the head of the release branch or cherry-picking
individual fixes, I can see a third option. Michael suggested to
upstream that they delay backporting features until said feature has
been part of a release to get more real-world exposure. That idea hasn't
gotten traction there, but we could still do something similar, and base
our snapshot on the date of the last release, cherry-picking later
commits in a more usual fashion.

Using that method, for the 2.35 branch we'd only have 4 commits to
examine for cherry-picking given the 2.37 release on Feb 1.

Would either that last option or even the direct use of the upstream
branch as micro-release updates be acceptable for an SRU?

Cheers,
--
Simon Chopin
Foundations Team                               	    Ubuntu MOTU/Core Dev
simon.chopin at canonical.com                            schopin at ubuntu.com