[Bug 1982534] Re: Quirks: Add upgrade quirk for FIPS enabled systems to work around libgcrypt hmac file placement

[Brian Murray]

** Tags added: foundations-todo

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubuntu-release-upgrader in
Ubuntu.
https://bugs.launchpad.net/bugs/1982534

Title:
  Quirks: Add upgrade quirk for FIPS enabled systems to work around
  libgcrypt hmac file placement

Status in ubuntu-release-upgrader package in Ubuntu:
  New
Status in ubuntu-release-upgrader source package in Focal:
  In Progress

Bug description:
  [Impact]

  When upgrading a Bionic fips (NOT fips-updates) system to Focal fips,
  the upgrade gets stuck generating the initramfs with the following
  message:

  update-initramfs: Generating /boot/initrd.img-5.4.0-84-generic
  Failed to copy HMAC file "/usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac".
  E: /usr/share/initramfs-tools/hooks/fips-libgcrypt failed with return 1.
  update-initramfs: failed for /boot/initrd.img-5.4.0-84-generic with 1.
  dpkg: error processing package ubuntu-fips (--configure):
   installed ubuntu-fips package post-installation script subprocess returned error exit status 1
   
  This happens because we are trying to copy /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to the initramfs, but because of a bug, it is actually installed to /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac.

  Most normal Focal systems are not affected, due to being usrmerged
  [1], where /lib is a symlink to /usr/lib.

  [1] https://wiki.debian.org/UsrMerge

  But when we upgrade from Bionic, usrmerge is not performed.

  We have resolved this bug in two new package releases for Focal, in fips-updates:
  - ubuntu-fips (1.2.4+updates1)
  - libgcrypt20 (1.8.5-5ubuntu1.fips.1.5)

  This was a part of bug 1944403, which is private due to having
  debdiffs for FIPS packages attached to the bug. Ask mruffell for
  access if you wish to view the bug.

  We changed /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be placed in
  /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac, and we changed the
  initramfs hook script to look for existence in both /usr/lib and /lib.

  The problem is, libgcrypt20 (1.8.5-5ubuntu1.fips.1.5) did not make the
  current re-certification round and Canonical have already received a
  certificate for libgcrypt20 (1.8.5-5ubuntu1.fips.1.4) stating fips
  compliance, thus we cannot get 1.8.5-5ubuntu1.fips.1.5 certified in
  this current round.

  The next certification round may take place in a year or two, but we
  require a solution for users who wish to upgrade from Bionic fips to
  Focal fips, and those users cannot enable fips-updates for strict
  compliance reasons.

  The workaround is to add a dpkg-divert, somewhat of the form of:

  $ [ ! -L /lib ] && sudo dpkg-divert --local --rename --add --divert
  /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac /lib/x86_64-linux-
  gnu/.libgcrypt.so.20.hmac

  I have implemented this as a upgrade quirk in DistUpgradeQuirks.py, in
  the function _fipsLibgcryptDivert(), which checks for the presence of
  libgcrypt20-hmac and checking to see if the broken fips certified
  package would be installed, before adding the dpkg-divert.

  Note, I did not use the --rename option in the quirk, as we require
  /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac to be left in place while
  we are performing the upgrade, as the libgcrypt self-test will fail if
  the file is removed before we install the Focal libgcrypt package. The
  error is very noisy as it is printed each time a program is invoked
  that uses libgcrypt, e.g. apt, dpkg, and it could prevent the upgrade
  from completing successfully.

  libgcrypt selftest: binary  (0): No such file or directory
  (/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac)

  Instead, we leave /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac in place
  until the very end of the upgrade, where we remove it in the function
  _fipsLibgcryptHmacCleanup().

  [Testcase]

  There are three upgrades that need to be tested to ensure things work
  as intended:

  - Bionic
  - Bionic fips
  - Bionic fips-updates

  Systems need to be attached to a UA token, and fips or fips-updates
  enabled and installed, before upgrade starts.

  You can use do-release-upgrade to perform the upgrade.

  The patch can be found here:

  https://paste.ubuntu.com/p/FgrnsRfZPt/

  You can test the upgrade before the merge to ubuntu-release-upgrader
  by:

  $ sudo -s
  # vim ~/quirk.patch
  Stick the contents of the pastebin in the file.

  # apt update
  # apt upgrade
  # do-release-upgrade

  At the first screen where you are asked to start another ssh server, hit ctrl-c
  and then press x to exit screen.

  We need to patch the quirk file we just downloaded.

  # cd /tmp
  # cd ubuntu-release-upgrader-<RANDOM LETTERS>
  # patch -p1 < ~/quirk.patch

  Hopefully it applies cleanly.

  We can restart do-release-upgrade with our patched copy with:

  # ./focal

  From there, all upgrades should succeed cleanly.

  Ideally we would also test upgrading to Jammy from the resulting Focal
  install, to ensure usrmerge works as intended, and that
  /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac was cleaned up correctly.

  [Where problems could occur]

  We are adding a quirk to Bionic systems upgrading to Focal only. This
  quirk is not intended to be used on any platform other than
  Bionic->Focal, and thus only targets the Focal branch of ubuntu-
  release-upgrader.

  We check to make sure that libgcrypt20-hmac is installed, and if the
  package is not installed, we exit from the quirk. This way we ensure
  we only apply the quirk to FIPS enabled systems.

  We also exit from the quirk if fips-updates is being used, and the
  fixed libgcrypt20-hmac package is to be installed as part of the
  upgrade.

  On cleanup, we take great care to ensure that we only remove the
  Bionic /lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file, and leave the
  Focal /usr/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac file well alone.
  We ensure the system is not usrmerged, and check the hashes of both
  files to make sure they do not match before removing the Bionic hmac
  file.

  If a regression were to occur, it could prevent systems from upgrading
  correctly, and would have a significant impact to users before it
  would be fixed.

  I believe I have narrowed the risk down to the small pool of fips
  users that require this quirk to be able to upgrade successfully.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1982534/+subscriptions