[Bug 2064096] Re: Services fail to start in noble deployed with TPM+FDE

[Nick Rosbrook]

The autopkgtest regressions were all resolved with retries.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/2064096

Title:
  Services fail to start in noble deployed with TPM+FDE

Status in apparmor package in Ubuntu:
  New
Status in cups package in Ubuntu:
  Confirmed
Status in rsyslog package in Ubuntu:
  Confirmed
Status in sssd package in Ubuntu:
  Confirmed
Status in systemd package in Ubuntu:
  Confirmed
Status in unbound package in Ubuntu:
  New
Status in apparmor source package in Noble:
  New
Status in cups source package in Noble:
  New
Status in rsyslog source package in Noble:
  New
Status in sssd source package in Noble:
  New
Status in systemd source package in Noble:
  Fix Committed
Status in unbound source package in Noble:
  New

Bug description:
  [Impact]

  On systems that have systemd in the initrd, after the switch root,
  services trying to access resources in /run (e.g. /run/systemd/notify)
  will get AppArmor denials. This is because as a part of the switch
  root, before the pivot_root(), the /run (and /proc, /sys, /dev) are
  "moved" with a bind mount. Hence the new /run has a different mount
  id, and AppArmor thinks that e.g. /run/systemd/notify is disconnected
  from the current mount tree.

  [Test Plan]

  The simplest way to test this is to use dracut on a classic Ubuntu
  system:

  1. Create a VM running Ubuntu 24.04 LTS. The virtualization implementation is not important.
  2. Install dracut, and then reboot.

  $ apt install -y dracut

  3. Once rebooted, verify that systemd did a switch root:

  $ journalctl -b --grep "Switching root"

  4. Check for rsyslog AppArmor denials:

  $ dmesg | grep rsyslog

  On an affected system, the denials will be present. With the patch,
  there should be no denials (or at least not related to accessing files
  in /run).

  [Where problems could occur]

  Using MS_MOVE rather than MS_BIND for /run during the switch root means that there is a brief time where /run (in the old root) is not available for units running before the pivot_root(). So, if we were to see problems, it would likely
  be related to problems with resources in /run, very close to the switch root timeframe. However, before noble, the switch root *is* done using MS_MOVE on /run (and /proc, /sys, and /dev), so have reasonable evidence that this is a safe change.

  [Other information]

  We only change the flags for /run because that is the filesystem that
  seems affected in practice. In particular, we leave /proc alone
  because code in systemd may use /proc between the time it is moved to
  the new root, but before the pivot_root(), which would be a riskier
  change.

  [Original Description]

  What's known so far:
  - 24.04 desktop deployed with TPM+FDE shows this bug
  - services confined with apparmor that need to access something in /run/systemd (like the notify socket) fail to do so, even if the apparmor profile is in complain mode. And the apparmor profile does already have rules to allow that access
  - only after running aa-disable <path> can the service start fine
  - paths logged by the apparmor DENIED or ALLOWED messages are missing the "/run" prefix from "/run/systemd/......".
  - When we add rules to the profile using "/systemd/...." (i.e., also dropping the /run prefix), then it works
  - other access in /run/systemd/ are also blocked, but the most noticeable one is the notify mechanism
  - comment #2 also states that azure CVM images are also impacted
  - comment #4 has instructions on how to create such a VM locally with LXD vms

  Original description follows:

  This might be related to #2064088

  The rsyslog service is continually timing out and restarting. If I use
  a service drop-in file and change the 'Type' from 'notify' to
  'simple', the service starts and appears to work normally.

  In the journal, I can see the attached apparmor errors. I can't make
  sense of them, but if it's a similar issue to #2064088, then I suspect
  apparmor is preventing the systemd notify function from alerting
  systemd that the service is up and running.

  ProblemType: Bug
  DistroRelease: Ubuntu 24.04
  Package: rsyslog 8.2312.0-3ubuntu9
  ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
  Uname: Linux 6.8.0-31-generic x86_64
  ApportVersion: 2.28.1-0ubuntu2
  Architecture: amd64
  CasperMD5CheckMismatches: ./boot/grub/grub.cfg
  CasperMD5CheckResult: fail
  CurrentDesktop: ubuntu:GNOME
  Date: Mon Apr 29 10:37:46 2024
  ProcEnviron:
   LANG=en_GB.UTF-8
   PATH=(custom, no user)
   SHELL=/bin/bash
   TERM=xterm-256color
   XDG_RUNTIME_DIR=<set>
  SourcePackage: rsyslog
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2064096/+subscriptions