[Bug 1982534] Re: Quirks: Add upgrade quirk for FIPS enabled systems to work around libgcrypt hmac file placement
Robie Basak
1982534 at bugs.launchpad.net
Wed Jan 4 17:21:18 UTC 2023
Some comments and questions on SRU review:
This is a bit special since it's not the usual "backport a fix" type
SRU, so I thought that reviewing the code changes themselves seemed more
important than usual in this case. So here are my code review comments:
Why are you calling subprocess.Popen() instead of
subprocess.check_call() or similar? To use Popen() directly, you also
need to call wait() on the output of subprocess.Popen() really and then
check the result code. It'll work if you don't, but any problems will
fail silently and won't be logged, so as written this seems buggy to me.
On check_call() vs. call(), I suggest check_call() since you're already
catching, logging and then ignoring errors, and a failure would then be
logged appropriately.
Presumably this script needs to work with the Python version shipped in
Bionic? I checked and that seems to be 3.6, and
https://docs.python.org/3/library/subprocess.html#subprocess.check_call
says check_call() was introduced prior to 3.3, so I think you're good
there.
Why are you invoking the rm command instead of calling os.unlink()?
Note that there are cases where two files are the same but not a
symlink. A hard link is one, and that would be fine in this case.
Another case that would not be fine is a bind mount. You might conclude
that this case isn't reasonable to support; I'll leave that decision to
you.
However, if you aren't checking hashes following review changes, then
please update the bug description so that you're no longer saying that
you are :)
** Changed in: ubuntu-release-upgrader (Ubuntu Focal)
Status: In Progress => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubuntu-release-upgrader in
Ubuntu.
https://bugs.launchpad.net/bugs/1982534
Title:
Quirks: Add upgrade quirk for FIPS enabled systems to work around
libgcrypt hmac file placement
Status in ubuntu-release-upgrader package in Ubuntu:
New
Status in ubuntu-release-upgrader source package in Focal:
Incomplete
Bug description:
[Impact]
When upgrading a Bionic fips (NOT fips-updates) system to Focal fips,
the upgrade gets stuck generating the initramfs with the following
message:
update-initramfs: Generating /boot/initrd.img-5.4.0-84-generic
Failed to copy HMAC file "/usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac".
E: /usr/share/initramfs-tools/hooks/fips-libgcrypt failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-84-generic with 1.
dpkg: error processing package ubuntu-fips (--configure):
installed ubuntu-fips package post-installation script subprocess returned error exit status 1
This happens because we are trying to copy /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to the initramfs, but because of a bug, it is actually installed to /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac.
Most normal Focal systems are not affected, due to being usrmerged
[1], where /lib is a symlink to /usr/lib.
[1] https://wiki.debian.org/UsrMerge
But when we upgrade from Bionic, usrmerge is not performed.
We have resolved this bug in two new package releases for Focal, in fips-updates:
- ubuntu-fips (1.2.4+updates1)
- libgcrypt20 (1.8.5-5ubuntu1.fips.1.5)
This was a part of bug 1944403, which is private due to having
debdiffs for FIPS packages attached to the bug. Ask mruffell for
access if you wish to view the bug.
We changed /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be placed in
/usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac, and we changed the
initramfs hook script to look for existence in both /usr/lib and /lib.
The problem is, libgcrypt20 (1.8.5-5ubuntu1.fips.1.5) did not make the
current re-certification round and Canonical have already received a
certificate for libgcrypt20 (1.8.5-5ubuntu1.fips.1.4) stating fips
compliance, thus we cannot get 1.8.5-5ubuntu1.fips.1.5 certified in
this current round.
The next certification round may take place in a year or two, but we
require a solution for users who wish to upgrade from Bionic fips to
Focal fips, and those users cannot enable fips-updates for strict
compliance reasons.
The workaround is to add a dpkg-divert, somewhat of the form of:
$ [ ! -L /lib ] && sudo dpkg-divert --local --rename --add --divert
/usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac /lib/x86_64-linux-
gnu/.libgcrypt.so.20.hmac
I have implemented this as a upgrade quirk in DistUpgradeQuirks.py, in
the function _fipsLibgcryptDivert(), which checks for the presence of
libgcrypt20-hmac and checking to see if the broken fips certified
package would be installed, before adding the dpkg-divert.
Note, I did not use the --rename option in the quirk, as we require
/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be left in place while
we are performing the upgrade, as the libgcrypt self-test will fail if
the file is removed before we install the Focal libgcrypt package. The
error is very noisy as it is printed each time a program is invoked
that uses libgcrypt, e.g. apt, dpkg, and it could prevent the upgrade
from completing successfully.
libgcrypt selftest: binary (0): No such file or directory
(/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac)
Instead, we leave /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac in place
until the very end of the upgrade, where we remove it in the function
_fipsLibgcryptHmacCleanup().
[Testcase]
There are three upgrades that need to be tested to ensure things work
as intended:
- Bionic
- Bionic fips
- Bionic fips-updates
Systems need to be attached to a UA token, and fips or fips-updates
enabled and installed, before upgrade starts.
You can use do-release-upgrade to perform the upgrade.
The patch can be found here:
https://paste.ubuntu.com/p/FgrnsRfZPt/
You can test the upgrade before the merge to ubuntu-release-upgrader
by:
$ sudo -s
# vim ~/quirk.patch
Stick the contents of the pastebin in the file.
# apt update
# apt upgrade
# do-release-upgrade
At the first screen where you are asked to start another ssh server, hit ctrl-c
and then press x to exit screen.
We need to patch the quirk file we just downloaded.
# cd /tmp
# cd ubuntu-release-upgrader-<RANDOM LETTERS>
# patch -p1 < ~/quirk.patch
Hopefully it applies cleanly.
We can restart do-release-upgrade with our patched copy with:
# ./focal
From there, all upgrades should succeed cleanly.
Ideally we would also test upgrading to Jammy from the resulting Focal
install, to ensure usrmerge works as intended, and that
/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac was cleaned up correctly.
[Where problems could occur]
We are adding a quirk to Bionic systems upgrading to Focal only. This
quirk is not intended to be used on any platform other than
Bionic->Focal, and thus only targets the Focal branch of ubuntu-
release-upgrader.
We check to make sure that libgcrypt20-hmac is installed, and if the
package is not installed, we exit from the quirk. This way we ensure
we only apply the quirk to FIPS enabled systems.
We also exit from the quirk if fips-updates is being used, and the
fixed libgcrypt20-hmac package is to be installed as part of the
upgrade.
On cleanup, we take great care to ensure that we only remove the
Bionic /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file, and leave the
Focal /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file well alone.
We ensure the system is not usrmerged, and check the hashes of both
files to make sure they do not match before removing the Bionic hmac
file.
If a regression were to occur, it could prevent systems from upgrading
correctly, and would have a significant impact to users before it
would be fixed.
I believe I have narrowed the risk down to the small pool of fips
users that require this quirk to be able to upgrade successfully.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1982534/+subscriptions
More information about the foundations-bugs
mailing list