[Bug 1590799] Re: nfs-kernel-server does not start because of dependency failure

Maarten Jacobs 1590799 at bugs.launchpad.net
Sun May 19 19:45:22 UTC 2019 

As it turns out, I'd not subscribed to this bug so I didn't get comments
after I posted mine. I had to refer back to my own comment to figure out
the solution since my NFS server apparently broke earlier this week.

An update to nfs-common (I think that's the package that installs the
service files) was pushed out and overwrote the "fix" I'd put in
earlier.

A more complete fix (after some trial and error) was to create the
"override.conf" files for nfs-server and nfs-mountd using:

sudo systemctl edit nfs-server (same for nfs-mountd).

I copy/pasted the baseline service definitions into the override files
and added my own tweaks. With that, the override files I am currently
running with are as follows (output from systemctl cat nfs-server and
nfs-mountd):

-------
# /etc/systemd/system/nfs-server.service.d/override.conf
[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires= network.target proc-fs-nfsd.mount
Requires= nfs-mountd.service
Wants=rpcbind.socket
Wants=nfs-idmapd.service

After= local-fs.target
After= network.target proc-fs-nfsd.mount rpcbind.socket nfs-mountd.service
After= nfs-idmapd.service rpc-statd.service
Before= rpc-statd-notify.service

# GSS services dependencies and ordering
Wants=auth-rpcgss-module.service
After=rpc-gssd.service rpc-svcgssd.service

# start/stop server before/after client
Before=remote-fs-pre.target

Wants=nfs-config.service
After=nfs-config.service

BindsTo=rpcbind.service
After=rpcbind.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils

ExecStart=
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/sbin/exportfs -r
ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
ExecStop=/usr/sbin/rpc.nfsd 0
ExecStopPost=/usr/sbin/exportfs -au
ExecStopPost=/usr/sbin/exportfs -f

ExecReload=/usr/sbin/exportfs -r

[Install]
WantedBy=multi-user.target
------

Note the addition of:
   BindsTo=rpcbind.service
   After=rpcbind.service
   ExecStart=

------
# /etc/systemd/system/nfs-mountd.service.d/override.conf
[Unit]
Description=NFS Mount Daemon
DefaultDependencies=no
Requires=proc-fs-nfsd.mount
After=proc-fs-nfsd.mount
After=network.target local-fs.target
After=rpcbind.socket
BindsTo=nfs-server.service

Wants=nfs-config.service rpcbind.service
After=nfs-config.service rpcbind.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils
Type=forking
ExecStart=
ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS
------

Note the addition of "rpcbind.service" on the "Wants" and "After" lines,
as well as the "ExecStart=".

It took me a bit to figure out the "ExecStart=" directives were
required. As I found out, the ExecStart directive is additive, so the
"ExecStart=" directive is required to "clear out" the directive set in
the baseline service files. (In other words, the baseline service file
is read before override.conf is read).

This appears to work for me on Ubuntu 16.04.06. I am not in a position
to verify these tweaks work on later versions of Ubuntu.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1590799

Title:
  nfs-kernel-server does not start because of dependency failure

Status in nfs-utils package in Ubuntu:
  Fix Released
Status in nfs-utils source package in Trusty:
  Invalid
Status in nfs-utils source package in Xenial:
  Fix Released
Status in nfs-utils source package in Yakkety:
  Invalid
Status in nfs-utils source package in Zesty:
  Fix Released

Bug description:
  [Impact]

   * nfs-mountd doesn't get started because of a race condition happening when rpcbind.socket is not specified as a needed service for it to start.
   * nfs-server using rpcbind.target instead of using rpcbind.socket. Target should not be used (Comment #24)

  [Test Case]

   * Install nfs-kernel-server inside a xenial lxc guest and restart it until nfs-mountd doesn't start complaining on rpc error.
   * Comment #25

  [Regression Potential]

   * Cons: Systemd dependencies could brake for nfs-server and nfs-mountd.
   * Pros: Patches have been accepted upstream (and tested).

  [Other Info]
   
  # Original Bug Description

  Immediately after boot:

  root at feynmann:~# systemctl status nfs-kernel-server
  ● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
     Active: inactive (dead)

  Jun 09 14:35:47 feynmann systemd[1]: Dependency failed for NFS server and services.
  Jun 09 14:35:47 feynmann systemd[1]: nfs-server.service: Job nfs-server.service/start failed

  root at feynmann:~# systemctl status nfs-mountd.service
  ● nfs-mountd.service - NFS Mount Daemon
     Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2016-06-09 14:35:47 BST; 7min ago
    Process: 1321 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=1/FAILURE)

  Jun 09 14:35:47 feynmann systemd[1]: Starting NFS Mount Daemon...
  Jun 09 14:35:47 feynmann rpc.mountd[1321]: mountd: could not create listeners
  Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Control process exited, code=exited
  Jun 09 14:35:47 feynmann systemd[1]: Failed to start NFS Mount Daemon.
  Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Unit entered failed state.
  Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.

  root at feynmann:~# systemctl list-dependencies nfs-kernel-server
  nfs-kernel-server.service
  ● ├─auth-rpcgss-module.service
  ● ├─nfs-config.service
  ● ├─nfs-idmapd.service
  ● ├─nfs-mountd.service
  ● ├─proc-fs-nfsd.mount
  ● ├─rpc-svcgssd.service
  ● ├─system.slice
  ● ├─network.target
  ● └─rpcbind.target
  ●   └─rpcbind.service

  root at feynmann:~# systemctl list-dependencies nfs-mountd.service
  nfs-mountd.service
  ● ├─nfs-config.service
  ● ├─nfs-server.service
  ● ├─proc-fs-nfsd.mount
  ● └─system.slice
  root at feynmann:~#

  root at feynmann:~# lsb_release -rd
  Description:	Ubuntu 16.04 LTS
  Release:	16.04

  root at feynmann:~# apt-cache policy nfs-kernel-server
  nfs-kernel-server:
    Installed: 1:1.2.8-9ubuntu12
    Candidate: 1:1.2.8-9ubuntu12
    Version table:
   *** 1:1.2.8-9ubuntu12 500
          500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
          100 /var/lib/dpkg/status

  Additional comments:

      1. There seems to be a circular dependency between nfs-mountd and nfs-kernel-server
      2. I can get it working by changing the AFter,Requires in /lib/ssystemd/system/nfs-{mountd|server}.service files. I have managed to get nfs-kernel-server to start but not nfs-mountd.
      3. /usr/lib/systemd/scripts/nfs-utils_env.sh references /etc/sysconfig/nfs which is Centos/RedHat location of this file. Also /etc/default/nfs does not exist. (possibly unrelated to this bug)
      4. A file "/lib/systemd/system/-.slice" exists. this file prevents execution of 'ls *' or 'grep xxx *' commands in that directory. I am unsure whether this is intended by the systemd developers but it is unfriendly when investigating this bug.

  Attempted solution:

      1. Edit /lib/systemd/system/nfs-server.service (original lines are
  commented out:

  [Unit]
  Description=NFS server and services
  DefaultDependencies=no
  Requires=network.target proc-fs-nfsd.mount rpcbind.target
  # Requires=nfs-mountd.service
  Wants=nfs-idmapd.service

  After=local-fs.target
  #After=network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
  After=network.target proc-fs-nfsd.mount rpcbind.target
  After=nfs-idmapd.service rpc-statd.service
  #Before=rpc-statd-notify.service
  Before=nfs-mountd.service rpc-statd-notify.service
  ...

  followed by a systemctl daemon-reload and a reboot.

  This results in nfs-kernel-server starting correctly but nfs-mountd
  not so. However starting nfs-mountd manually after reboot is
  successful:

  root at feynmann:~# systemctl status nfs-kernel-server.service
  ● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2016-06-09 15:07:23 BST; 1min 25s ago
    Process: 1391 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
    Process: 1384 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
   Main PID: 1391 (code=exited, status=0/SUCCESS)
      Tasks: 0
     Memory: 0B
        CPU: 0
     CGroup: /system.slice/nfs-server.service

  Jun 09 15:07:23 feynmann systemd[1]: Starting NFS server and services...
  Jun 09 15:07:23 feynmann systemd[1]: Started NFS server and services.

  root at feynmann:~# systemctl status nfs-mountd.service
  ● nfs-mountd.service - NFS Mount Daemon
     Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
     Active: inactive (dead)

  root at feynmann:~# systemctl start nfs-mountd.service

  root at feynmann:~# systemctl status nfs-mountd.service
  ● nfs-mountd.service - NFS Mount Daemon
     Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
     Active: active (running) since Thu 2016-06-09 15:09:02 BST; 3s ago
    Process: 2044 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
   Main PID: 2046 (rpc.mountd)
      Tasks: 1
     Memory: 904.0K
        CPU: 12ms
     CGroup: /system.slice/nfs-mountd.service
             └─2046 /usr/sbin/rpc.mountd --manage-gids

  Jun 09 15:09:02 feynmann systemd[1]: Starting NFS Mount Daemon...
  Jun 09 15:09:02 feynmann rpc.mountd[2046]: Version 1.2.8 starting
  Jun 09 15:09:02 feynmann systemd[1]: Started NFS Mount Daemon.

  Enabling nfs-mountd.service (systemctl enable nfs-mountd.service) has
  no effect in this case.

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: nfs-kernel-server 1:1.2.8-9ubuntu12 [modified: lib/systemd/system/nfs-server.service]
  ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
  Uname: Linux 4.4.0-22-generic x86_64
  ApportVersion: 2.20.1-0ubuntu2.1
  Architecture: amd64
  Date: Thu Jun  9 14:38:58 2016
  InstallationDate: Installed on 2016-06-08 (1 days ago)
  InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
  ProcEnviron:
   SHELL=/bin/bash
   TERM=linux
   PATH=(custom, no user)
   LANG=en_GB.UTF-8
   LANGUAGE=en_GB:en
  SourcePackage: nfs-utils
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1590799/+subscriptions

More information about the foundations-bugs mailing list