[Bug 1590799] Re: nfs-kernel-server does not start because of dependency failure
Rafael David Tinoco
1590799 at bugs.launchpad.net
Thu Aug 20 17:52:40 UTC 2020
This issue also does NOT affect Bionic:
[rafaeldtinoco at bnfstests ~]$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2020-08-20 17:46:54 UTC; 29s ago
Process: 1537 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
Process: 1536 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
Process: 1535 ExecStop=/usr/sbin/rpc.nfsd 0 (code=exited, status=0/SUCCESS)
Process: 1561 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Process: 1560 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Main PID: 1561 (code=exited, status=0/SUCCESS)
Aug 20 17:46:54 bnfstests systemd[1]: Starting NFS server and services...
Aug 20 17:46:54 bnfstests systemd[1]: Started NFS server and services.
[rafaeldtinoco at bnfstests ~]$ systemctl status rpcbind.socket
Failed to dump process list, ignoring: No such file or directory
● rpcbind.socket - RPCbind Server Activation Socket
Loaded: loaded (/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-08-20 17:44:25 UTC; 3min 6s ago
Listen: /run/rpcbind.sock (Stream)
CGroup: /system.slice/rpcbind.socket
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[rafaeldtinoco at bnfstests ~]$ systemctl status rpcbind.service
● rpcbind.service - RPC bind portmap service
Loaded: loaded (/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-08-20 17:44:26 UTC; 3min 14s ago
Docs: man:rpcbind(8)
Main PID: 382 (rpcbind)
Tasks: 1 (limit: 2338)
CGroup: /system.slice/rpcbind.service
└─382 /sbin/rpcbind -f -w
Aug 20 17:44:26 bnfstests systemd[1]: Starting RPC bind portmap service...
Aug 20 17:44:26 bnfstests systemd[1]: Started RPC bind portmap service.
[rafaeldtinoco at bnfstests ~]$ systemctl status nfs-
nfs-blkmap.service nfs-config.service nfs-mountd.service
nfs-client.target nfs-idmapd.service nfs-server.service
nfs-common.service nfs-kernel-server.service nfs-utils.service
[rafaeldtinoco at bnfstests ~]$ systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
Active: active (running) since Thu 2020-08-20 17:46:54 UTC; 54s ago
Process: 1556 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
Main PID: 1559 (rpc.mountd)
Tasks: 1 (limit: 2338)
CGroup: /system.slice/nfs-mountd.service
└─1559 /usr/sbin/rpc.mountd --manage-gids
Aug 20 17:46:54 bnfstests systemd[1]: Starting NFS Mount Daemon...
Aug 20 17:46:54 bnfstests rpc.mountd[1559]: Version 1.3.3 starting
Aug 20 17:46:54 bnfstests systemd[1]: Started NFS Mount Daemon.
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-blkmap.service
disabled
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-client.target
enabled
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-common.service
masked
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-config.service
static
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-idmapd.service
static
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-kernel-server.service
enabled
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-mountd.service
static
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-server.service
enabled
[rafaeldtinoco at bnfstests ~]$ systemctl is-enabled nfs-utils.service
static
[rafaeldtinoco at bnfstests ~]$ systemctl is-active rpcbind
active
[rafaeldtinoco at bnfstests ~]$ systemctl is-active rpcbind.service
active
[rafaeldtinoco at bnfstests ~]$ systemctl is-active rpcbind.socket
active
[rafaeldtinoco at bnfstests ~]$ systemctl is-active nfs-kernel-server.service
active
[rafaeldtinoco at bnfstests ~]$ systemctl is-active nfs-mountd.service
active
[rafaeldtinoco at bnfstests ~]$ systemctl is-active nfs-client.target
active
Note: if it affects your Bionic is very likely that you are using:
[rafaeldtinoco at bnfstests ~]$ ls /etc/init.d/*nfs*
/etc/init.d/nfs-common /etc/init.d/nfs-kernel-server
[rafaeldtinoco at bnfstests ~]$ ls /etc/init.d/*rpc*
/etc/init.d/rpcbind
systemd generators using the services from /etc/init.d/ (you can check
that using systemctl status XXXX). If you are using only systemd NFS
service units, then you should NOT get " DEPENDENCY ERRORS " (the
original situation described by this bug).
Of course there are OTHER issues that can happen during NFS service
initialization (or any of its dependent services). Feel free to open NEW
bugs if you think you faced a bug (and not a misconfiguration issue).
For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/community
(or in the existing mailing lists)
** Changed in: nfs-utils (Ubuntu Bionic)
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1590799
Title:
nfs-kernel-server does not start because of dependency failure
Status in nfs-utils package in Ubuntu:
Fix Released
Status in nfs-utils source package in Xenial:
Fix Released
Status in nfs-utils source package in Zesty:
Fix Released
Status in nfs-utils source package in Bionic:
Fix Released
Bug description:
NOTE FOR THIS BUG (READ THIS BEFORE POSTING)
Whoever finds this and thinks is facing this same problem, please, be
aware that disabling all NFS related SysV init scripts (/etc/init.d)
is advised before posting here you are also suffering from this same
issue.
There was an ORIGINAL ISSUE with *systemd units* and NOT the units
that are created automatically by systemd because of existing SysV
(/etc/init.d) files being still enabled.
Having enabled units that are automatically generated by systemd on
behalf of /etc/init.d (specially after upgrades) can indeed cause
dependency issues and the fix for this is to have ONLY systemd units
enabled for NFS services.
If not totally understand, completely uninstall nfs-kernel-server and
related packages, purge configs, install nfs-kernel-server package
again and that will make sure systemd units are used by default.
------------
[Impact]
* nfs-mountd doesn't get started because of a race condition happening when rpcbind.socket is not specified as a needed service for it to start.
* nfs-server using rpcbind.target instead of using rpcbind.socket. Target should not be used (Comment #24)
[Test Case]
* Install nfs-kernel-server inside a xenial lxc guest and restart it until nfs-mountd doesn't start complaining on rpc error.
* Comment #25
[Regression Potential]
* Cons: Systemd dependencies could brake for nfs-server and nfs-mountd.
* Pros: Patches have been accepted upstream (and tested).
[Other Info]
# Original Bug Description
Immediately after boot:
root at feynmann:~# systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Jun 09 14:35:47 feynmann systemd[1]: Dependency failed for NFS server and services.
Jun 09 14:35:47 feynmann systemd[1]: nfs-server.service: Job nfs-server.service/start failed
root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2016-06-09 14:35:47 BST; 7min ago
Process: 1321 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=1/FAILURE)
Jun 09 14:35:47 feynmann systemd[1]: Starting NFS Mount Daemon...
Jun 09 14:35:47 feynmann rpc.mountd[1321]: mountd: could not create listeners
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Control process exited, code=exited
Jun 09 14:35:47 feynmann systemd[1]: Failed to start NFS Mount Daemon.
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Unit entered failed state.
Jun 09 14:35:47 feynmann systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.
root at feynmann:~# systemctl list-dependencies nfs-kernel-server
nfs-kernel-server.service
● ├─auth-rpcgss-module.service
● ├─nfs-config.service
● ├─nfs-idmapd.service
● ├─nfs-mountd.service
● ├─proc-fs-nfsd.mount
● ├─rpc-svcgssd.service
● ├─system.slice
● ├─network.target
● └─rpcbind.target
● └─rpcbind.service
root at feynmann:~# systemctl list-dependencies nfs-mountd.service
nfs-mountd.service
● ├─nfs-config.service
● ├─nfs-server.service
● ├─proc-fs-nfsd.mount
● └─system.slice
root at feynmann:~#
root at feynmann:~# lsb_release -rd
Description: Ubuntu 16.04 LTS
Release: 16.04
root at feynmann:~# apt-cache policy nfs-kernel-server
nfs-kernel-server:
Installed: 1:1.2.8-9ubuntu12
Candidate: 1:1.2.8-9ubuntu12
Version table:
*** 1:1.2.8-9ubuntu12 500
500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
Additional comments:
1. There seems to be a circular dependency between nfs-mountd and nfs-kernel-server
2. I can get it working by changing the AFter,Requires in /lib/ssystemd/system/nfs-{mountd|server}.service files. I have managed to get nfs-kernel-server to start but not nfs-mountd.
3. /usr/lib/systemd/scripts/nfs-utils_env.sh references /etc/sysconfig/nfs which is Centos/RedHat location of this file. Also /etc/default/nfs does not exist. (possibly unrelated to this bug)
4. A file "/lib/systemd/system/-.slice" exists. this file prevents execution of 'ls *' or 'grep xxx *' commands in that directory. I am unsure whether this is intended by the systemd developers but it is unfriendly when investigating this bug.
Attempted solution:
1. Edit /lib/systemd/system/nfs-server.service (original lines are
commented out:
[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires=network.target proc-fs-nfsd.mount rpcbind.target
# Requires=nfs-mountd.service
Wants=nfs-idmapd.service
After=local-fs.target
#After=network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
After=network.target proc-fs-nfsd.mount rpcbind.target
After=nfs-idmapd.service rpc-statd.service
#Before=rpc-statd-notify.service
Before=nfs-mountd.service rpc-statd-notify.service
...
followed by a systemctl daemon-reload and a reboot.
This results in nfs-kernel-server starting correctly but nfs-mountd
not so. However starting nfs-mountd manually after reboot is
successful:
root at feynmann:~# systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2016-06-09 15:07:23 BST; 1min 25s ago
Process: 1391 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Process: 1384 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Main PID: 1391 (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 0B
CPU: 0
CGroup: /system.slice/nfs-server.service
Jun 09 15:07:23 feynmann systemd[1]: Starting NFS server and services...
Jun 09 15:07:23 feynmann systemd[1]: Started NFS server and services.
root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
Active: inactive (dead)
root at feynmann:~# systemctl start nfs-mountd.service
root at feynmann:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
Active: active (running) since Thu 2016-06-09 15:09:02 BST; 3s ago
Process: 2044 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
Main PID: 2046 (rpc.mountd)
Tasks: 1
Memory: 904.0K
CPU: 12ms
CGroup: /system.slice/nfs-mountd.service
└─2046 /usr/sbin/rpc.mountd --manage-gids
Jun 09 15:09:02 feynmann systemd[1]: Starting NFS Mount Daemon...
Jun 09 15:09:02 feynmann rpc.mountd[2046]: Version 1.2.8 starting
Jun 09 15:09:02 feynmann systemd[1]: Started NFS Mount Daemon.
Enabling nfs-mountd.service (systemctl enable nfs-mountd.service) has
no effect in this case.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: nfs-kernel-server 1:1.2.8-9ubuntu12 [modified: lib/systemd/system/nfs-server.service]
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Thu Jun 9 14:38:58 2016
InstallationDate: Installed on 2016-06-08 (1 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
ProcEnviron:
SHELL=/bin/bash
TERM=linux
PATH=(custom, no user)
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
SourcePackage: nfs-utils
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1590799/+subscriptions
More information about the foundations-bugs
mailing list