[Bug 1979952] Re: Bind mount to NFS mount fails on Ubuntu 22.04

Nick Rosbrook 1979952 at bugs.launchpad.net
Mon Sep 12 19:32:02 UTC 2022 

I have verified this fix using systemd 249.11-0ubuntu3.6 from jammy-
proposed:

nr at clean-jammy-amd64:~$ apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.6
  Candidate: 249.11-0ubuntu3.6
  Version table:
 *** 249.11-0ubuntu3.6 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.4 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
nr at clean-jammy-amd64:~$ cat /etc/fstab 
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/vda3 during installation
UUID=32a6b55f-64d3-421b-a77d-2f9a2c6d68be /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/vda2 during installation
UUID=C591-26A1  /boot/efi       vfat    umask=0077      0       1
/swapfile                                 none            swap    sw              0       0
172.16.99.107:/data /data nfs defaults,nfsvers=3 0 0
/data/home /home2 none bind,_netdev,x-systemd.requires-mounts-for=/data 0 0
nr at clean-jammy-amd64:~$ systemctl status home2.mount
● home2.mount - /home2
     Loaded: loaded (/etc/fstab; generated)
     Active: active (mounted) since Mon 2022-09-12 15:29:19 EDT; 29s ago
      Where: /home2
       What: 172.16.99.107:/data/home
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)
      Tasks: 0 (limit: 1084)
     Memory: 16.0K
        CPU: 2ms
     CGroup: /system.slice/home2.mount

Sep 12 15:29:19 clean-jammy-amd64 systemd[1]: Mounting /home2...
Sep 12 15:29:19 clean-jammy-amd64 systemd[1]: Mounted /home2.


** Tags removed: verification-needed verification-needed-jammy
** Tags added: verification-done verification-done-jammy

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1979952

Title:
  Bind mount to NFS mount fails on Ubuntu 22.04

Status in systemd:
  Unknown
Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Jammy:
  Fix Committed
Status in systemd source package in Kinetic:
  Fix Released

Bug description:
  [Impact]
  Some NFS setups that have worked on previous versions of Ubuntu are now broken. Specifically, when creating a bind mount, systemd will fail if the directory already exists on NFS.

  [Test Plan]
  This test plan requires an NFS server to be in place. The exact setup is not important, but for the purposes of this test plan we will assume the server exports /data, which contains a directory called home.

  * Make sure nfs-common is installed:

   $ apt install nfs-common -y

  * Check the NFS server exports using showmount:

   $ showmount -e $NFS_SERVER_IP
   Export list for $NFS_SERVER_IP:
   /data *

  * Add the appropriate entries to /etc/fstab. In this example, we will
  make /home2 a bind mount to /data/home:

   # /etc/fstab
   $NFS_SERVER_IP:/data /data nfs defaults,nfsvers=3 0 0
   /data/home /home2 none bind,_netdev,x-systemd.requires-mounts-for=/data 0 0

  * Check systemctl status home2.mount. On an affected system, we should
  see a permission denied error:

   $ systemctl status home2.mount
   x home2.mount - /home2
        Loaded: loaded (/etc/fstab; generated)
        Active: failed (Result: resources)
         Where: /home2
          What: /data/home
          Docs: man:fstab(5)
                man:systemd-fstab-generator(8)

   systemd[1]: home2.mount: Failed to make bind mount source '/data/home': Permission denied
   systemd[1]: home2.mount: Failed to run 'mount' task: Permission denied
   systemd[1]: home2.mount: Failed with result 'resources'.
   systemd[1]: Failed to mount /home2.

  * On a patched system, we expect /home2 to be mounted successfully.

  [Where problems could occur]
  The patch makes it so that systemd will not fail bind mounts immediately if creating the directory fails. Thus, any regressions would be seen during bind mounts with systemd. N.B. that this patch restores previous behavior in systemd.

  [Original Description]
  I have the following setup:
  - /data is an NFS mount.
  - /home is a bind mount to /data/home.

  Configured in /etc/fstab with these lines:

  $FILE_SERVER:/data /data nfs defaults 0 0
  /data/home /home none bind,_netdev,x-systemd.requires-mounts-for=/data 0 0

  This has worked with the same configuration for me on at least Ubuntu 18.04 and Ubuntu 20.04, but on Ubuntu 22.04 the mount of /home fails when attempted by systemd.
  So not only is /home not mounted after boot, but also when I run "sudo systemctl start home.mount", it fails.

  The journal entries are this:

  sudo[1316]:  wendler : TTY=pts/0 ; PWD=/home/wendler ; USER=root ; COMMAND=/usr/bin/systemctl start home.mount
  sudo[1316]: pam_unix(sudo:session): session opened for user root(uid=0) by wendler(uid=1000)
  systemd[1]: home.mount: Directory /home to mount over is not empty, mounting anyway.
  systemd[1]: home.mount: Failed to make bind mount source '/data/home': Permission denied
  systemd[1]: home.mount: Failed to run 'mount' task: Permission denied
  systemd[1]: home.mount: Failed with result 'resources'.
  systemd[1]: Failed to mount /home.

  However, when I run "sudo mount /home" it works.

  Now the weird thing is that after I have mounted /home manually once
  and unmounted it again, then "sudo systemctl start home.mount"
  suddenly also works! But of course only until the next reboot.

  And even if I just do "ls /data" once (either as root or as my user),
  it also makes "sudo systemctl start home.mount" start working!

  To be fully clear:

  - Booting the system (/home fails to mount)
  - sudo systemctl start home.mount  # fails
  - sudo ls /data  # shows correct output
  - sudo systemctl start home.mount  # works!

  Additional information:

  The NFS export for /data has root_squash set and if I remove this
  option, the bind mount of /home works as it should. However, both
  /data and /data/home have mode rwxr-xr-x, so root is able to enter and
  read these directories even despite root_squash.

  It is not a network or mount-order problem. Not only is the bind mount
  to /home correctly attempted after /data is mounted during boot,
  remember that the bind mount also fails when I try it with "sudo
  systemctl start home.mount" minutes afterwards.

  Versions (system is a recent installation with all updates applied):
  - Ubuntu 22.04
  - Linux 5.15.0-40-generic
  - systemd 249.11-0ubuntu3.3
  - util-linux 2.37.2-4ubuntu3

  This appears on both Ubuntu 22.04 machines that I have (a hardware
  machine and a VM).

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1979952/+subscriptions

More information about the foundations-bugs mailing list