[Bug 2091737] Re: Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from 1:1.3.4-2.5ubuntu3)

Mon Dec 16 18:49:34 UTC 2024

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: nfs-utils (Ubuntu)
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/2091737

Title:
  Slow NFS Mounts with nfs-utils 1:2.6.1-1ubuntu1.2 (Regression from
  1:1.3.4-2.5ubuntu3)

Status in nfs-utils package in Ubuntu:
  Confirmed

Bug description:
  We are experiencing significant delays (60+ seconds) when mounting NFS
  directories after exporting 40 directories. This issue is observed
  with nfs-utils 1:2.6.1-1ubuntu1.2 on an Ubuntu Jammy system. We have
  40 exports and are attempting to mount via NFS onto /sd/mount paths.
  The /sd filesystem is a custom, in-house filesystem built on top of
  FUSE. Prior to using nfs-utils 1:2.6.1-1ubuntu1.2, we were using nfs-
  utils 1:1.3.4-2.5ubuntu3(Ubuntu 20.04 focal) where mounts were almost
  immediate, taking less than 10 seconds.

  Applying this patch from upstream nfs-utils resolves the issue:
  https://patchwork.kernel.org/project/linux-
  nfs/patch/20231023021052.5258-4-neilb at suse.de/

  Additionally, the above problem is not reproducible using nfs-utils
  1:2.6.4-3ubuntu5(Ubuntu 24.04 Noble Numbat)

  System Information:
  OS: Ubuntu 22.04 LTS Jammy
  nfs-utils Version: 1:2.6.1-1ubuntu1.2
  Kernel: 5.15.0-117

  Steps to reproduce internally:
  Create 40 directories in /sd/mount and export them.
  Execute showmount -e to list the exports.
  Attempt to mount one of the NFS directories and note the time taken.

  Actual Outcome:
  Mounting a single directory takes more than 60 seconds consistently. While attempting to perform multiple mounts, some mounts can take more than 300s.

  Expected Outcome:
  Mounting should be significantly faster.

  Logs and Outputs:
  Export List:
  ubuntu at vm-machine-s82gbm-3ojtydo:~$ showmount -e
  Export list for vm-machine-s82gbm-3ojtydo:
  /sd/mount/oracle/jammy_patch/test40 *
  /sd/mount/oracle/jammy_patch/test39 *
  /sd/mount/oracle/jammy_patch/test38 *
  /sd/mount/oracle/jammy_patch/test37 *
  /sd/mount/oracle/jammy_patch/test36 *
  /sd/mount/oracle/jammy_patch/test35 *
  /sd/mount/oracle/jammy_patch/test34 *
  /sd/mount/oracle/jammy_patch/test33 *
  /sd/mount/oracle/jammy_patch/test32 *
  /sd/mount/oracle/jammy_patch/test31 *
  /sd/mount/oracle/jammy_patch/test30 *
  /sd/mount/oracle/jammy_patch/test29 *
  /sd/mount/oracle/jammy_patch/test28 *
  /sd/mount/oracle/jammy_patch/test27 *
  /sd/mount/oracle/jammy_patch/test26 *
  /sd/mount/oracle/jammy_patch/test25 *
  /sd/mount/oracle/jammy_patch/test24 *
  /sd/mount/oracle/jammy_patch/test23 *
  /sd/mount/oracle/jammy_patch/test22 *
  /sd/mount/oracle/jammy_patch/test21 *
  /sd/mount/oracle/jammy_patch/test20 *
  /sd/mount/oracle/jammy_patch/test19 *
  /sd/mount/oracle/jammy_patch/test18 *
  /sd/mount/oracle/jammy_patch/test17 *
  /sd/mount/oracle/jammy_patch/test16 *
  /sd/mount/oracle/jammy_patch/test15 *
  /sd/mount/oracle/jammy_patch/test14 *
  /sd/mount/oracle/jammy_patch/test13 *
  /sd/mount/oracle/jammy_patch/test12 *
  /sd/mount/oracle/jammy_patch/test11 *
  /sd/mount/oracle/jammy_patch/test10 *
  /sd/mount/oracle/jammy_patch/test9  *
  /sd/mount/oracle/jammy_patch/test8  *
  /sd/mount/oracle/jammy_patch/test7  *
  /sd/mount/oracle/jammy_patch/test6  *
  /sd/mount/oracle/jammy_patch/test5  *
  /sd/mount/oracle/jammy_patch/test4  *
  /sd/mount/oracle/jammy_patch/test3  *
  /sd/mount/oracle/jammy_patch/test2  *
  /sd/mount/oracle/jammy_patch/test1  *
  /sd/mount                           ::1

  Mount command:
  time sudo mount -o rw,hard,rsize=1048576,wsize=1048576,vers=3,nointr,tcp,sync ${SERVER_IP}:/sd/mount/oracle/jammy_patch/test1 /tmp/test/test1

  Patch Applied:
  Fix commit: https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/
  diff --git a/utils/exportd/exportd.c b/utils/exportd/exportd.c
  index 2dd12cb6015b..6f866445efc2 100644
  --- a/utils/exportd/exportd.c
  +++ b/utils/exportd/exportd.c
  @@ -289,12 +289,16 @@ main(int argc, char **argv)
  else if (num_threads > MAX_THREADS)
  num_threads = MAX_THREADS;
  /* Open cache channel files BEFORE forking so each upcall is
  only handled by one thread. Kernel provides locking for both
  read and write.
  */
  cache_open();
  if (num_threads > 1) fork_workers();
  /* Open files now to avoid sharing descriptors among forked processes */
  cache_open();
  v4clients_init();
  /* Process incoming upcalls */
  diff --git a/utils/mountd/mountd.c b/utils/mountd/mountd.c
  index bcf749fabbb3..f9c62cded66c 100644
  --- a/utils/mountd/mountd.c
  +++ b/utils/mountd/mountd.c
  @@ -916,12 +916,16 @@ main(int argc, char **argv)
  else if (num_threads > MAX_THREADS)
  num_threads = MAX_THREADS;
  /* Open cache channel files BEFORE forking so each upcall is
  only handled by one thread. Kernel provides locking for both
  read and write.
  */
  cache_open();
  if (num_threads > 1)
  fork_workers();
  nfsd_path_init();
  /* Open files now to avoid sharing descriptors among forked processes */
  cache_open();
  v4clients_init();
  xlog(L_NOTICE, "Version " VERSION " starting");

  The above patch seems to fix the following commit which is present on
  1:2.6.1-1ubuntu1.2 : https://patchwork.kernel.org/project/linux-
  nfs/patch/20190521124701.61849-2-trond.myklebust at hammerspace.com/

  Questions:
  1. Is the mount issue relevant to the above patch?
  2. We are not yet prepared to upgrade to Ubuntu Noble yet which has the patch of interest. Can we expect another upgrade for nfs-utils on Jammy which incorporates this patch?
  3. Can the patch  https://patchwork.kernel.org/project/linux-nfs/patch/20231023021052.5258-4-neilb@suse.de/ be applied in isolation?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/2091737/+subscriptions