[Bug 1950787] [NEW] systemd-sysusers cannot mount /dev in privileged containers (to pass credentials)

Lukas Märdian 1950787 at bugs.launchpad.net
Fri Nov 12 13:46:20 UTC 2021 

Public bug reported:

systemd-sysusers.service/systemd.exec fails to start in privileged containers, due to being unable to properly mount /dev for passing credentials, caused by the following config in the .service unit:
```
# Optionally, pick up a root password and shell for the root user from a
# credential passed to the service manager. This is useful for importing this
# data from nspawn's --set-credential= switch.
LoadCredential=passwd.hashed-password.root
LoadCredential=passwd.plaintext-password.root
LoadCredential=passwd.shell.root
```

Reproducer:
$ lxc profile set default security.privileged "true"
$ lxc launch ubuntu-daily:jammy test
$ lxc exec test bash
# add-apt-repository ppa:ci-train-ppa-service/4704
# apt install systemd # install systemd 249.5-2ubuntu1
# systemctl restart systemd-sysusers
# systemctl status systemd-sysusers
# system --status=failed
$ lxc profile set default security.privileged "false"

A workaround is to disable it via:
$ cat /etc/systemd/system/systemd-sysusers.service.d/override.conf:
[Service]
LoadCredential=

Interesting logs:
Nov 12 12:09:44 test systemd[1]: systemd-journald.service: Added fd 42 (n/a) to fd store.
Nov 12 12:09:44 test systemd[431]: Mounting /dev (MS_REC|MS_SLAVE "")...
Nov 12 12:09:44 test systemd[431]: Failed to mount n/a (type n/a) on /dev (MS_REC|MS_SLAVE ""): Permission denied
Nov 12 12:09:44 test systemd[430]: (sd-mkdcreds) failed with exit status 1.
Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed to set up credentials: Protocol error
Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed at step CREDENTIALS spawning

** Affects: lxd (Ubuntu)
     Importance: Undecided
         Status: New

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: New

** Also affects: systemd (Ubuntu)
   Importance: Undecided
       Status: New

** Description changed:

  systemd-sysusers.service/systemd.exec fails to start in privileged containers, due to being unable to properly mount /dev for passing credentials, caused by the following config in the .service unit:
  # Optionally, pick up a root password and shell for the root user from a
  # credential passed to the service manager. This is useful for importing this
  # data from nspawn's --set-credential= switch.
  LoadCredential=passwd.hashed-password.root
  LoadCredential=passwd.plaintext-password.root
  LoadCredential=passwd.shell.root
  
  Reproducer:
  $ lxc profile set default security.privileged "true"
  $ lxc launch ubuntu-daily:jammy test
  $ lxc exec test bash
  # add-apt-repository ppa:ci-train-ppa-service/4704
  # apt install systemd # install systemd 249.5-2ubuntu1
  # systemctl restart systemd-sysusers
  # systemctl status systemd-sysusers
  # system --status=failed
  $ lxc profile set default security.privileged "false"
  
  A workaround is to disable it via:
  $ cat /etc/systemd/system/systemd-sysusers.service.d/override.conf:
  [Service]
  LoadCredential=
  
  Interesting logs:
  Nov 12 12:09:44 test systemd[1]: systemd-journald.service: Added fd 42 (n/a) to fd store.
  Nov 12 12:09:44 test systemd[431]: Mounting /dev (MS_REC|MS_SLAVE "")...
  Nov 12 12:09:44 test systemd[431]: Failed to mount n/a (type n/a) on /dev (MS_REC|MS_SLAVE ""): Permission denied
  Nov 12 12:09:44 test systemd[430]: (sd-mkdcreds) failed with exit status 1.
  Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed to set up credentials: Protocol error
  Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed at step CREDENTIALS spawning
- 
- Debug logs:
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Job 350 systemd-sysusers.service/restart finished, result=done
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Converting job systemd-sysusers.service/restart -> systemd-sysusers.service/start
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: ConditionNeedsUpdate=/etc succeeded.
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Passing 0 fds to service
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: About to execute systemd-sysusers
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Forked systemd-sysusers as 430
- Nov 12 12:09:44 test systemd[430]: Successfully forked off '(sd-mkdcreds)' as PID 431.
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=7 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=8 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2893 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2894 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Changed failed -> start
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=9 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2895 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Starting Create System Users...
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=10 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=11 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2896 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dsysusers_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2897 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/job/350 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=12 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/job/350 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2898 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: systemd-journald.service: Got notification message from PID 59 (FDSTORE=1)
- Nov 12 12:09:44 test systemd[1]: systemd-journald.service: Added fd 42 (n/a) to fd store.
- Nov 12 12:09:44 test systemd[431]: Mounting /dev (MS_REC|MS_SLAVE "")...
- Nov 12 12:09:44 test systemd[431]: Failed to mount n/a (type n/a) on /dev (MS_REC|MS_SLAVE ""): Permission denied
- Nov 12 12:09:44 test systemd[430]: (sd-mkdcreds) failed with exit status 1.
- Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed to set up credentials: Protocol error
- Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed at step CREDENTIALS spawning systemd-sysusers: Protocol error
- Nov 12 12:09:44 test systemd[1]: Received SIGCHLD from PID 430 ((sysusers)).
- Nov 12 12:09:44 test systemd[1]: Child 430 ((sysusers)) died (code=exited, status=243/CREDENTIALS)
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Child 430 belongs to systemd-sysusers.service.
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Main process exited, code=exited, status=243/CREDENTIALS
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Failed with result 'exit-code'.
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Service will not restart (restart setting)
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Changed start -> failed
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=13 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2899 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
- Nov 12 12:09:44 test systemd[1]: systemd-sysusers.service: Job 350 systemd-sysusers.service/start finished, result=failed
- Nov 12 12:09:44 test systemd[1]: Failed to start Create System Users.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1950787

Title:
  systemd-sysusers cannot mount /dev in privileged containers (to pass
  credentials)

Status in lxd package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  New

Bug description:
  systemd-sysusers.service/systemd.exec fails to start in privileged containers, due to being unable to properly mount /dev for passing credentials, caused by the following config in the .service unit:
  ```
  # Optionally, pick up a root password and shell for the root user from a
  # credential passed to the service manager. This is useful for importing this
  # data from nspawn's --set-credential= switch.
  LoadCredential=passwd.hashed-password.root
  LoadCredential=passwd.plaintext-password.root
  LoadCredential=passwd.shell.root
  ```

  Reproducer:
  $ lxc profile set default security.privileged "true"
  $ lxc launch ubuntu-daily:jammy test
  $ lxc exec test bash
  # add-apt-repository ppa:ci-train-ppa-service/4704
  # apt install systemd # install systemd 249.5-2ubuntu1
  # systemctl restart systemd-sysusers
  # systemctl status systemd-sysusers
  # system --status=failed
  $ lxc profile set default security.privileged "false"

  A workaround is to disable it via:
  $ cat /etc/systemd/system/systemd-sysusers.service.d/override.conf:
  [Service]
  LoadCredential=

  Interesting logs:
  Nov 12 12:09:44 test systemd[1]: systemd-journald.service: Added fd 42 (n/a) to fd store.
  Nov 12 12:09:44 test systemd[431]: Mounting /dev (MS_REC|MS_SLAVE "")...
  Nov 12 12:09:44 test systemd[431]: Failed to mount n/a (type n/a) on /dev (MS_REC|MS_SLAVE ""): Permission denied
  Nov 12 12:09:44 test systemd[430]: (sd-mkdcreds) failed with exit status 1.
  Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed to set up credentials: Protocol error
  Nov 12 12:09:44 test systemd[430]: systemd-sysusers.service: Failed at step CREDENTIALS spawning

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1950787/+subscriptions

More information about the foundations-bugs mailing list