[Bug 1089941] [NEW] rpc.statd causes NFS clients to hang while doing hostname lookups

[Andrew Phillips]

Public bug reported:

It appears that rpc.statd from nfs-utils 1.2.5-3ubuntu3.1  will cause
clients to hang if the resolution of the IP address of a client takes
several seconds.

We've recently deployed Ubuntu 12.04 NFS servers and shortly after installation clients were hanging.   At the time, the server was logging messages like:
Dec 11 20:06:19 nfs-server kernel: [18392.347264] statd: server rpc.statd not responding, timed out
Dec 11 20:06:19 nfs-server kernel: [18392.347298] lockd: cannot monitor client1
Dec 11 20:06:54 nfs-server kernel: [18427.364941] statd: server rpc.statd not responding, timed out
Dec 11 20:06:54 nfs-server kernel: [18427.364972] lockd: cannot monitor client2
Dec 11 20:07:29 nfs-server kernel: [18462.382624] statd: server rpc.statd not responding, timed out
Dec 11 20:07:29 nfs-server kernel: [18462.382654] lockd: cannot monitor client3

I did some stracing of rpc.statd and noticed that it was talking to
avahi and then hanging for several seconds.  Apparently mdns lookups
through avahai have a timeout of 5 seconds.   It turns out that a few
client systems didn't have working reverse DNS and was causing mdns4 to
kick in to attempt to resolve them.  The host without the reverse DNS
was on a different subnet so it's not surprising that avahi couldn't
find it.

At the time our nsswitch.conf had:
hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4

After disabling mdns4 and restarting statd the problem did not reoccur.  mdns4_minimal does not exhibit the problem as it filters out most IP addresses:
hosts:          files mdns4_minimal [NOTFOUND=return] dns

I suspect that if other directory services (DNS, LDAP, whatever) are
slow to return results for the hosts table that it will behave in the
same way.

** Affects: nfs-utils (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1089941

Title:
  rpc.statd causes NFS clients to hang while doing hostname lookups

Status in “nfs-utils” package in Ubuntu:
  New

Bug description:
  It appears that rpc.statd from nfs-utils 1.2.5-3ubuntu3.1  will cause
  clients to hang if the resolution of the IP address of a client takes
  several seconds.

  We've recently deployed Ubuntu 12.04 NFS servers and shortly after installation clients were hanging.   At the time, the server was logging messages like:
  Dec 11 20:06:19 nfs-server kernel: [18392.347264] statd: server rpc.statd not responding, timed out
  Dec 11 20:06:19 nfs-server kernel: [18392.347298] lockd: cannot monitor client1
  Dec 11 20:06:54 nfs-server kernel: [18427.364941] statd: server rpc.statd not responding, timed out
  Dec 11 20:06:54 nfs-server kernel: [18427.364972] lockd: cannot monitor client2
  Dec 11 20:07:29 nfs-server kernel: [18462.382624] statd: server rpc.statd not responding, timed out
  Dec 11 20:07:29 nfs-server kernel: [18462.382654] lockd: cannot monitor client3

  I did some stracing of rpc.statd and noticed that it was talking to
  avahi and then hanging for several seconds.  Apparently mdns lookups
  through avahai have a timeout of 5 seconds.   It turns out that a few
  client systems didn't have working reverse DNS and was causing mdns4
  to kick in to attempt to resolve them.  The host without the reverse
  DNS was on a different subnet so it's not surprising that avahi
  couldn't find it.

  At the time our nsswitch.conf had:
  hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4

  After disabling mdns4 and restarting statd the problem did not reoccur.  mdns4_minimal does not exhibit the problem as it filters out most IP addresses:
  hosts:          files mdns4_minimal [NOTFOUND=return] dns

  I suspect that if other directory services (DNS, LDAP, whatever) are
  slow to return results for the hosts table that it will behave in the
  same way.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1089941/+subscriptions