[Bug 1959047] Re: systemd ignores RootDirectory option in .service units

Tue Mar 29 21:08:05 UTC 2022

Hello MegaBrutal, or anyone else affected,

Accepted lxd into bionic-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/lxd/3.0.3-0ubuntu1~18.04.2 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
bionic to verification-done-bionic. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-bionic. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: lxd (Ubuntu Bionic)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-bionic

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1959047

Title:
  systemd ignores RootDirectory option in .service units

Status in lxd package in Ubuntu:
  Invalid
Status in systemd package in Ubuntu:
  Confirmed
Status in lxd source package in Bionic:
  Fix Committed
Status in systemd source package in Bionic:
  Confirmed
Status in systemd source package in Focal:
  Confirmed
Status in systemd source package in Impish:
  Confirmed

Bug description:
  [Impact]

  Ubuntu carries a patch on top of systemd [a] to silence
  namespace set up failures. This is meant as a workaround
  for a bug in the LXD version shipped in Ubuntu 18.04.

  Masking namespace set up failures creates a false sense of
  security for the user/admin.

  As mentioned in comment #1, systemd upstream explains that silencing
  this kind of error is dangerous and should be avoided.

  Backporting the LXD fix [b] to Ubuntu 18.04 would allow namespaces
  to work inside containers. This is the goal of this SRU.

  Ultimately, once LXD in Ubuntu 18.04 includes the fix [b], it would
  be possible to drop the Ubuntu-specific patch for systemd [a]. This
  is however *not an immediate concern for this SRU*.

  [Test Plan]

  1) Create a 18.04 VM:
  $ lxc launch images:ubuntu/18.04 lp1959047 --vm
  $ sleep 30  # give it time to boot

  2) Install and initialize LXD in it:
  $ lxc exec lp1959047 -- apt-get update
  $ lxc exec lp1959047 -- apt-get install -y lxd
  $ lxc exec lp1959047 -- lxd init --auto

  3) Create a Jammy container and enable systemd debugging:
  $ lxc exec lp1959047 -- lxc init images:ubuntu/22.04 c1
  $ lxc exec lp1959047 -- lxc config set c1 raw.lxc 'lxc.init.cmd = /sbin/init systemd.log_level=debug'
  $ lxc exec lp1959047 -- lxc start c1

  4) Check if namespace set up failures are logged:
  $ lxc exec lp1959047 --  lxc exec c1 -- journalctl -b0 --grep 'Failed to set up namespace'
  Mar 24 23:29:19 c1 systemd[99]: systemd-udevd.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:19 c1 systemd[132]: systemd-networkd.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:19 c1 systemd[131]: systemd-logind.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:20 c1 systemd[136]: systemd-resolved.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:20 c1 systemd[128]: e2scrub_reap.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied
  Mar 24 23:29:23 c1 systemd[243]: systemd-hostnamed.service: Failed to set up namespace, assuming containerized execution, ignoring: Permission denied

  If LXD in Ubuntu 18.04 has the patch, the "Failed to set up namespace"
  messages wouldn't be there.

  [Where problems could occur]

  The LXD fix changes the Apparmor profile used for containers. This essentially
  loosen the mount restrictions applied to containers.

  Weakening the Apparmor profile could make it easier for a process in the container
  to do damage that would have otherwise been blocked. On the other hand, this
  allows making use of namespaces/sandboxing inside the container.

  Upstream LXD has the fix since 2019 which make it less likely to run into
  problems with the backport.

  The backported fix was also tested manually to ensure LXD still behaved normally
  and that it avoided the namespace set up failures in Jammy containers.

  [a]: https://git.launchpad.net/ubuntu/+source/systemd/tree/debian/patches/debian/UBUNTU-Revert-namespace-be-more-careful-when-handling-namespacin.patch?h=ubuntu/jammy
  [b]: https://github.com/lxc/lxd/commit/a6b780703350faff8328f3d565f6bac7b6dcf59f

  [Initial bug description]

  The version of systemd (249.5-2ubuntu4) currently packaged for the
  Ubuntu development version (22.04 Jammy Jellyfish) totally ignores the
  RootDirectory= option in systemd service files. With RootDirectory,
  systemd should start the service after calling chroot() on the
  supplied directory.

  To test/reproduce, create a test service file with the following
  contents:

  # /etc/systemd/system/lsb-release.service
  [Unit]
  Description=LSB Release Information

  [Service]
  Type=simple
  RootDirectory=/var/chroot/trusty
  ExecStartPre=/bin/pwd
  ExecStart=/usr/bin/lsb_release -a

  You should have a chroot environment in the specified RootDirectory,
  even though you can still deduce if systemd attempted to chroot or not
  from the resulting error message.

  In my example, I installed an end-of-life Ubuntu 14.04 Trusty Tahr in
  the chroot environment. On systems NOT affected by the problem, I get
  the following result when I start this test service. This is what I'd
  expect.

  Jan 25 20:40:40 dolly systemd[1]: Starting LSB Release Information...
  Jan 25 20:40:40 dolly pwd[361]: /
  Jan 25 20:40:40 dolly systemd[1]: Started LSB Release Information.
  Jan 25 20:40:40 dolly lsb_release[362]: No LSB modules are available.
  Jan 25 20:40:40 dolly lsb_release[362]: Distributor ID:        Ubuntu
  Jan 25 20:40:40 dolly lsb_release[362]: Description:        Ubuntu 14.04 LTS
  Jan 25 20:40:40 dolly lsb_release[362]: Release:        14.04
  Jan 25 20:40:40 dolly lsb_release[362]: Codename:        trusty
  Jan 25 20:40:40 dolly systemd[1]: lsb-release.service: Succeeded.

  On the problematic system, however, I get the following result.

  Jan 25 21:21:08 savelog systemd[1]: Starting LSB Release Information...
  Jan 25 21:21:08 savelog systemd[1]: Started LSB Release Information.
  Jan 25 21:21:08 savelog pwd[81114]: /
  Jan 25 21:21:08 savelog lsb_release[81115]: No LSB modules are available.
  Jan 25 21:21:08 savelog lsb_release[81115]: Distributor ID:        Ubuntu
  Jan 25 21:21:08 savelog lsb_release[81115]: Description:        Ubuntu Jammy Jellyfish (development branch)
  Jan 25 21:21:08 savelog lsb_release[81115]: Release:        22.04
  Jan 25 21:21:08 savelog lsb_release[81115]: Codename:        jammy
  Jan 25 21:21:08 savelog systemd[1]: lsb-release.service: Deactivated successfully.

  It totally run the service on the host's root filesystem, it didn't
  care even the slightest that a RootDirectory is specified.

  Tested on the following releases / systemd versions:

  Ubuntu 18.04.6 Bionic Beaver – ISSUE NOT PRESENT
  systemd 237
  +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

  Ubuntu 20.04.3 Focal Fossa – ISSUE NOT PRESENT
  systemd 245 (245.4-4ubuntu3.15)
  +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

  Ubuntu 21.10 Impish Indri – ISSUE NOT PRESENT
  systemd 248 (248.3-1ubuntu8.2)
  +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

  Ubuntu 22.04 Jammy Jellyfish (development branch) – ISSUE PRESENT
  systemd 249 (249.5-2ubuntu4)
  +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

  Note that the problem is produced under an LXC container; since
  systemd detects virtualization, it might change how it behaves.

  It's either a bug or an intentional change I don't understand yet
  (i.e. the RootDirectory option has deprecated and is about to be
  replaced with something else, or there are additional conditions to be
  met before RootDirectory is considered), but I think in the latter
  case I should at least get a warning that there is a change in
  configuration. I imagine suddenly everyone's existing service units
  utilizing RootDirectory silently stop working without any information
  regarding why.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1959047/+subscriptions