[Bug 1918141] Re: nfs-server.service needs name resolution and network online

Tue May 11 12:58:28 UTC 2021

Guys, this has been sitting in unapproved for about 2 months now, and I
need to push another fix for nfs-utils asap (see
https://bugs.launchpad.net/bugs/1918141). I only noticed this unapproved
upload when I was about to push the git tag, as the upload doesn't show
up in rmadison's output.

I would prefer to not include this fix here in my upload, as it looks
like there is no consensus and it's a bigger behavior change, and my is
a pure bugfix with upstream patches that are already applied in redhat,
debian, and focal+.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1918141

Title:
  nfs-server.service needs name resolution and network online

Status in nfs-utils package in Ubuntu:
  Fix Released
Status in nfs-utils source package in Bionic:
  Confirmed
Status in nfs-utils source package in Focal:
  Confirmed
Status in nfs-utils source package in Groovy:
  Confirmed
Status in nfs-utils source package in Hirsute:
  Fix Released

Bug description:
  [ Impact ]

  nfs-server.service (part of the nfs-utils package) has insufficient
  dependencies to start correctly in a setting where the NFS exports
  list contains DNS hostnames (not in local hosts file) or netgroups
  served via network (for example sssd).

  This issue can negatively impact users that, for example, cannot
  easily resort to using IP addresses directly in the /etc/exports file,
  and need to use hostnames instead in order to be able to mount NFS
  shares over the network.

  [ Test Case ]

  You can reproduce the issue by following the instructions below.
  Adjust the Ubuntu release on "images:ubuntu/hirsute" accordingly.

  $ lxc launch images:ubuntu/hirsute nfs-utils-bug1918141 --vm
  $ lxc shell nfs-utils-bug1918141
  # apt update
  # apt install -y nfs-kernel-server
  # mkdir /testshare
  # cat >> /etc/exports << EOF
  /testshare ubuntu.com(ro,sync,no_subtree_check)
  EOF
  # systemctl edit --full systemd-networkd.service

  You will have to:

  - Comment out the "Wants=" line.
  - Remove "network.target" from the "Before=" line.
  - Add a new "After=fakenet.service" line.
  - Add a new "Wants=fakenet.service" line.
  - Add the following line in the "[Service]" section:

    ExecStartPre=/bin/sleep 10

  # systemctl edit --force --full fakenet.service

  This new service file will contain:

  [Unit]
  Description=Fake network.target
  DefaultDependencies=no
  After=systemd-udevd.service network-pre.target systemd-sysusers.service systemd-sysctl.service
  Before=network.target multi-user.target shutdown.target
  Conflicts=shutdown.target
  Wants=network.target

  [Service]
  Type=oneshot
  ExecStart=/bin/true

  [Install]
  WantedBy=multi-user.target

  # systemctl enable fakenet.service
  # systemctl disable systemd-resolved
  # systemctl mask systemd-resolved
  # rm /etc/resolv.conf
  # cat > /etc/resolv.conf << EOF
  nameserver 8.8.8.8
  EOF
  # reboot

  After a few seconds, you can enter the VM again:

  $ lxc shell nfs-utils-bug1918141
  # sleep 20 && systemctl status nfs-server.service
  ● nfs-server.service - NFS server and services
       Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
       Active: active (exited) since Tue 2021-03-16 20:51:42 UTC; 39s ago
      Process: 310 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
      Process: 311 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
     Main PID: 311 (code=exited, status=0/SUCCESS)

  Mar 16 20:51:41 nfs-utils-bug1918141 systemd[1]: Starting NFS server and services...
  Mar 16 20:51:41 nfs-utils-bug1918141 exportfs[310]: exportfs: Failed to resolve ubuntu.com
  Mar 16 20:51:41 nfs-utils-bug1918141 exportfs[310]: exportfs: Failed to resolve ubuntu.com
  Mar 16 20:51:42 nfs-utils-bug1918141 systemd[1]: Finished NFS server and services.

  As you can see, the service was started but exportfs failed to resolve
  the hostname.

  [Where problems could occur]

  This fix has been applied upstream for quite a while now, and is even
  part of a release, so it has been extensively tested by users and
  other distributions that have more recent nfs-utils.  I cannot easily
  envision problems with the proposed changes but:

  * If the user has manually modified the nfs-server.service file
  without taking proper precautions (i.e., editin the file directly on
  /lib/systemd/system/ instead of using "systemctl edit"), then he might
  experience a conflict when installing a new version of the file, and
  his modifications will be lost.  However, this falls under the "local
  configuration issue", in my opinion.

  [ Original Description ]

  nfs-server.service has insufficient dependencies to start correctly in
  a setting where the nfs exports list contains DNS host names (not in
  local hosts file) or netgroups served via network (for example sssd).

  Typical failures listed by systemctl status nfs-server.service are:

  Mar 08 14:16:52 server.example.com exportfs[844]: exportfs: Failed to resolve client1.example.com
  Mar 08 14:16:52 server.example.com exportfs[844]: exportfs: Failed to resolve client2.example.com

  Our workaround is to add the appropriate dependecies in
  /etc/systemd/system/nfs-server.service.d/dependencies.conf like so:

  ----------------------8<----------------------------
  [Unit]
  # nfs-server.service runs exportfs on startup, thus we need to be able to
  # do host and netgroup lookups which requires network to be online.
  After=network-online.target nss-lookup.target nss-user-lookup.target
  ----------------------8<----------------------------

  While nfs-server.service do depend on network.target, that only means
  that the network has been configured. On physical hardware it can take
  significantly longer for the network to come online (8+ seconds for
  our 10G NICs). Also note that we configure static IP:s via systemd-
  networkd, things might behave differently when using DHCP, network-
  manager etc. In any case, depending on network.target is almost always
  wrong, and network-online.target is usually the right one.

  nss-lookup.target is needed to ensure that DNS resolution works, and
  nss-user-lookup.target is the best approximation to ensure that
  netgroup resolution via sssd or equivalent works. Usually things "just
  works" even without these dependencies, but to ensure correct startup
  they should be present.

  It should be noted that this seems to once have been fixed in Ubuntu,
  but has been lost along the way for quite some time. When googling I
  find for example https://code.launchpad.net/~ubuntu-
  branches/ubuntu/wily/nfs-utils/wily-201507271018/+merge/265946 that
  fixes the network.target vs network-online.target dependency, but it
  has since been lost in the wind it seems.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1918141/+subscriptions