[Bug 2066990] Re: openssl fails with out of memory messages while trying to load the FIPS provider in a non-FIPS container on a FIPS host

[Adrien Nader]

My plan is to make the library write the following error message when
failing to load a provider (it will apply to any provider):

    While loading "fips" provider: error:07880025:common libcrypto
routines::reason(37)

There was already an error but it was silent until it reached the
application which could then assume something inappropriate as we've
seen here.

There are a couple constraints.

First, I'm not loading error strings because I'm changing the inside of
the library and if they haven't been loaded already, the error message
will say "37" rather than "ERR_R_DSO_LIB"/"ERR_LIB_DSO". I plan to work
around that by expanding the title of this bug report so that it's easy
to find with a search engine. (this might also be because I'm relying on
an internal static buffer for formatting because I prefer to avoid
adding memory management in an error path)

The other constraint is that I need to "fprintf(stderr, ..." which is a
bit uncommon for openssl code but I don't want to pop an error from the
error stack like `ERR_print_errors{,_fp}()` does since I'm in the
library code and not in the application code. This means the only API
usable is `ERR_peek_last_error()`. Not a big issue but that makes for
some slightly alien-looking code.

All in all, the output for the `openssl rehash certs` from above will
contain the following text, once per cert found (i.e. 147 times here):

    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)
    out of memory

And I hope that one can copy-paste the new error message in a search
engine and end directly on this page.

Any thought on this before I integrate this? Rough timeline is that I
push that to plucky very soon (tomorrow maybe) and then SRU it in Noble
right after (but SRUs take time and this will probably be a staged
change until a new openssl security update is released).

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssl in Ubuntu.
https://bugs.launchpad.net/bugs/2066990

Title:
  openssl fails with out of memory messages while trying to load the
  FIPS provider in a non-FIPS container on a FIPS host

Status in ca-certificates package in Ubuntu:
  Invalid
Status in openssl package in Ubuntu:
  Confirmed

Bug description:
  I wanted to try the new Ubuntu 24.04 Noble Numbat based .NET docker
  image and updated the base docker image in our CI pipeline to
  mcr.microsoft.com/dotnet/sdk:8.0-noble. However, it results in an out-
  of-memory exception. Based on my investigation, the exception occurs
  specifically when the update-ca-certificates command is executed. I
  can also repro the issue with ubuntu:noble image which means it's not
  specific to .NET docker images. It works fine with Jammy, by the way.
  The problem likely lies with the Noble base image rather than the .NET
  image. I'm not sure what changes were made between Jammy and Noble,
  but it appears that updating certificates consumes a lot of memory in
  Noble. I adjusted some memory settings in our GitLab runner, but it
  didn't resolve the issue. I attached all Gitlab Runner shell logs for
  .NET 8 Jammy, .NET 8 Noble and Ubuntu Noble images.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/2066990/+subscriptions