[Bug 1999275] Re: systemd-binfmt.service fails in tests-in-lxd

[Nick Rosbrook]

** Changed in: systemd (Ubuntu Jammy)
       Status: Triaged => In Progress

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1999275

Title:
  systemd-binfmt.service fails in tests-in-lxd

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Jammy:
  In Progress

Bug description:
  [Impact]

  systemd-binfmt.service fails in LXD because access
  /proc/sys/binfmt_misc is prevented by LXD apparmor config. This causes
  our autopkgtest to fail, and leaves systemd-binfmt.service in a failed
  state:

  root at jammy:~# systemctl status systemd-binfmt
  × systemd-binfmt.service - Set Up Additional Binary Formats
       Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static)
       Active: failed (Result: exit-code) since Thu 2023-08-17 18:54:18 UTC; 1min 12s ago
         Docs: man:systemd-binfmt.service(8)
               man:binfmt.d(5)
               https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
               https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
     Main PID: 118 (code=exited, status=1/FAILURE)
          CPU: 5ms

  Aug 17 18:54:18 jammy systemd[1]: Starting Set Up Additional Binary Formats...
  Aug 17 18:54:18 jammy systemd-binfmt[118]: Failed to add binary format: Permission denied
  Aug 17 18:54:18 jammy systemd[1]: systemd-binfmt.service: Main process exited, code=exited, status=1/FAILURE
  Aug 17 18:54:18 jammy systemd[1]: systemd-binfmt.service: Failed with result 'exit-code'.
  Aug 17 18:54:18 jammy systemd[1]: Failed to start Set Up Additional Binary Formats.

  [Test Plan]

  1. test_no_failed from boot-and-services should pass.
  2. Manual test:

  * Start a Jammy LXD container:

  $ lxc launch ubuntu-daily:jammy jammy
  $ lxc exec jammy bash

  * In the container, check systemd-binfmt.service:
  root at jammy:~# systemctl status systemd-binfmt
  ● systemd-binfmt.service - Set Up Additional Binary Formats
       Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static)
      Drop-In: /etc/systemd/system/systemd-binfmt.service.d
               └─override.conf
       Active: active (exited) since Thu 2023-08-17 19:01:00 UTC; 2min 27s ago
         Docs: man:systemd-binfmt.service(8)
               man:binfmt.d(5)
               https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
               https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
     Main PID: 122 (code=exited, status=0/SUCCESS)
          CPU: 5ms

  Aug 17 19:01:00 jammy systemd[1]: Starting Set Up Additional Binary Formats...
  Aug 17 19:01:00 jammy systemd[1]: Finished Set Up Additional Binary Formats.
  root at jammy:~# journalctl -t systemd-binfmt -b
  Aug 17 19:01:00 jammy systemd-binfmt[122]: /proc/sys/fs/binfmt_misc is not mounted in read-write mode, skipping.

  [ Where problems could occur]

  The patches are contained to binfmt logic in systemd, and so any
  regressions would be within systemd-binfmt itself. The patches make it
  so that systemd-binfmt checks if /proc/sys/binfmt_misc is mounted as
  rw before trying to make any changes. If it is mounted, but read-only,
  then systemd-binfmt will exit cleanly and log a message indicating so.
  If the patch was incorrect, it could make it so that systemd-binfmt is
  not fully executed on systems where it should be.

  [Original Description]

  The systemd-binfmt.service requires read-write access to
  /proc/sys/fs/binfmt_misc, but this is not possible in unprivileged LXD
  containers without binfmt_misc namespace support in the kernel [1].
  When this service is triggered, we get the following failure from
  test_no_failed within tests-in-lxd:

  No failed units ... -------- journal for failed service systemd-binfmt.service -----------
  Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd[1]: Starting Set Up Additional Binary Formats...
  Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd-binfmt[99]: Failed to flush binfmt_misc rules, ignoring: Permission denied
  Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd-binfmt[99]: /usr/lib/binfmt.d/python3.10.conf:1: Failed to delete rule 'python3.10', ignoring: Permission denied
  Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd-binfmt[99]: /usr/lib/binfmt.d/python3.10.conf:1: Failed to add binary format 'python3.10': Permission denied
  Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd[1]: systemd-binfmt.service: Main process exited, code=exited, status=1/FAILURE
  Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd[1]: systemd-binfmt.service: Failed with result 'exit-code'.
  Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd[1]: Failed to start Set Up Additional Binary Formats.
  FAIL
  test_rsyslog (__main__.ServicesTest) ... ok
  test_tmp_cleanup (__main__.ServicesTest) ... ok
  test_tmp_mount (__main__.ServicesTest) ... ok
  test_udev (__main__.ServicesTest) ... skipped 'udev does not work in containers'

  ======================================================================
  FAIL: test_no_failed (__main__.ServicesTest)
  No failed units
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/tmp/autopkgtest.METFVA/build.LjB/real-tree/debian/tests/boot-and-services", line 74, in test_no_failed
      self.assertEqual(failed, [])
  AssertionError: Lists differ: ['systemd-binfmt.service loaded failed failed Set Up Additional Binary Formats'] != []

  First list contains 1 additional elements.
  First extra element 0:
  'systemd-binfmt.service loaded failed failed Set Up Additional Binary Formats'

  - ['systemd-binfmt.service loaded failed failed Set Up Additional Binary Formats']
  + []

  The systemd-binfmt.service is only started if there is a configuration
  to be loaded from one of the valid configuration directories (e.g.
  /usr/lib/binfmt.d/python3.10.conf). Previously, no such file was
  installed in the test LXD images, but a recent upload of python3.10
  added support for systemd-binfmt [2].

  [1] https://discuss.linuxcontainers.org/t/3-1-0-binfmt-support-service-in-unprivileged-guest-requires-write-access-on-hosts-proc-sys-fs-binfmt-misc/5249
  [2] https://launchpadlibrarian.net/632281245/python3.10_3.10.8-1_3.10.8-3.diff.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1999275/+subscriptions