[Bug 1466217] [NEW] unmounting NFS filesystem hangs on readlink even with lazy unmount

[John Gilmore]

Public bug reported:

This error occurs on an NFS client machine when one of its NFS servers stops responding (e.g. is powered-off).  The umount command provides a -l (lazy) option that is supposed to disconnect the mount point from the system so that no future commands that access the file system will hang due to the unresponsive NFS server.  This is supposed to work even when the NFS server is not responding.  The problem is that a sub-library used by the umount command is doing a readlink() on the filesystem, which of
course causes a hang before umount can actually unmount the filesystem.

The umount program uses a helper program called /sbin/umount.nfs (which
is a symlink to /sbin/mount.nfs), and that's where the bug lies.  When
you do:

  umount -l /images

and /images is an NFS mount, umount invokes:

  /sbin/umount.nfs /images -l

and /sbin/umount.nfs (which is part of the nfs-common package) does the
readlink, which can easily be verified by running it under strace -f.
Here is a GDB backtrace of /sbin/umount.nfs when it hangs:

(gdb) run /images -l
Starting program: /sbin/umount.nfs /images -l
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Breakpoint 1, readlink () at ../sysdeps/unix/syscall-template.S:81
81	../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  readlink () at ../sysdeps/unix/syscall-template.S:81
#1  0xb7768a0a in ?? () from /lib/i386-linux-gnu/libmount.so.1
#2  0xb7756073 in mnt_resolve_path () from /lib/i386-linux-gnu/libmount.so.1
#3  0xb7762887 in ?? () from /lib/i386-linux-gnu/libmount.so.1
#4  0xb77666f0 in mnt_context_prepare_umount ()
   from /lib/i386-linux-gnu/libmount.so.1
#5  0x0804afc1 in ?? ()
#6  0xb75b9a83 in __libc_start_main (main=0x804ad10, argc=3, argv=0xbfcc2e34, 
    init=0x8057b40, fini=0x8057bb0, rtld_fini=0xb77cc180 <_dl_fini>, 
    stack_end=0xbfcc2e2c) at libc-start.c:287
#7  0x0804b4fc in ?? ()

The actual readlink() call seems to occur in libmount, from a function under mnt_resolve_path.  If, under GDB, I cause
the readlink function to artificially return -1 rather than do the system call, the rest of the program succeeds in
lazily unmounting the hung filesystem.

I don't know if the proper fix is for /sbin/umount.nfs to avoid calling mnt_resolve_path, or to pass it a parameter that says, "Don't touch that filesystem while trying to resolve the path!!!".  I will leave that to the maintainers.  All I know is that it hangs
forever if you let the readlink system call occur, but it does the job it's supposed to do if you breakpoint at the readlink and
do "return (int)-1" and "continue" in the debugger.

** Affects: ubiquity (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubiquity in Ubuntu.
https://bugs.launchpad.net/bugs/1466217

Title:
  unmounting NFS filesystem hangs on readlink even with lazy unmount

Status in ubiquity package in Ubuntu:
  New

Bug description:
  This error occurs on an NFS client machine when one of its NFS servers stops responding (e.g. is powered-off).  The umount command provides a -l (lazy) option that is supposed to disconnect the mount point from the system so that no future commands that access the file system will hang due to the unresponsive NFS server.  This is supposed to work even when the NFS server is not responding.  The problem is that a sub-library used by the umount command is doing a readlink() on the filesystem, which of
  course causes a hang before umount can actually unmount the filesystem.

  The umount program uses a helper program called /sbin/umount.nfs
  (which is a symlink to /sbin/mount.nfs), and that's where the bug
  lies.  When you do:

    umount -l /images

  and /images is an NFS mount, umount invokes:

    /sbin/umount.nfs /images -l

  and /sbin/umount.nfs (which is part of the nfs-common package) does
  the readlink, which can easily be verified by running it under strace
  -f.  Here is a GDB backtrace of /sbin/umount.nfs when it hangs:

  (gdb) run /images -l
  Starting program: /sbin/umount.nfs /images -l
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

  Breakpoint 1, readlink () at ../sysdeps/unix/syscall-template.S:81
  81	../sysdeps/unix/syscall-template.S: No such file or directory.
  (gdb) bt
  #0  readlink () at ../sysdeps/unix/syscall-template.S:81
  #1  0xb7768a0a in ?? () from /lib/i386-linux-gnu/libmount.so.1
  #2  0xb7756073 in mnt_resolve_path () from /lib/i386-linux-gnu/libmount.so.1
  #3  0xb7762887 in ?? () from /lib/i386-linux-gnu/libmount.so.1
  #4  0xb77666f0 in mnt_context_prepare_umount ()
     from /lib/i386-linux-gnu/libmount.so.1
  #5  0x0804afc1 in ?? ()
  #6  0xb75b9a83 in __libc_start_main (main=0x804ad10, argc=3, argv=0xbfcc2e34, 
      init=0x8057b40, fini=0x8057bb0, rtld_fini=0xb77cc180 <_dl_fini>, 
      stack_end=0xbfcc2e2c) at libc-start.c:287
  #7  0x0804b4fc in ?? ()

  The actual readlink() call seems to occur in libmount, from a function under mnt_resolve_path.  If, under GDB, I cause
  the readlink function to artificially return -1 rather than do the system call, the rest of the program succeeds in
  lazily unmounting the hung filesystem.

  I don't know if the proper fix is for /sbin/umount.nfs to avoid calling mnt_resolve_path, or to pass it a parameter that says, "Don't touch that filesystem while trying to resolve the path!!!".  I will leave that to the maintainers.  All I know is that it hangs
  forever if you let the readlink system call occur, but it does the job it's supposed to do if you breakpoint at the readlink and
  do "return (int)-1" and "continue" in the debugger.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1466217/+subscriptions