[Bug 1071978] Re: nfs kernel server is very slow and causing high cpu load

[Dave Gilbert]

OK, lets see what we've got; if I'm reading this vmstat correctly it's split pretty
much between system time and wait (for IO):


procs -----------memory----------                ---swap--        -----io----           -system-- ----cpu----
 r  b   swpd   free         buff     cache       si   so    bi         bo        in      cs  us  sy id wa
 5  1  43964 120664 346700 2267880    0    0    17          18       16     13    2   0 97   0
 0  0  43964 192872 343108 2216144    0    0   178   79081 13851 18234  2 30 45 23
 1  1  43968 321924 339260 2051828    0    2   937   78577 14664 21679  3 34 33 30
 1  4  43968 144504 329904 2311564    0    0   124 104639 21611 34747  3 46   8 44
 1  2  43968 145264 319808 2323488    0    0   328 108663 24725 40773  3 39 19 40
 1  4  43968 146860 317844 2325192    0    0   140 183331 38848 64357  2 46   4 47

now the way I'm reading that it's shifting about 100-180MB/s - (the bo column); and reading your logs I think that's
to a spinny disc, so that's a perfectly respectable transfer rate; I wouldn't have actually expected more than 100MB/s
though (i.e. 1000Mbps) assuming gigabit ether; what type of transfer were you doing at the time - large file or lots of small
ones?

The 'wait' time doesn't worry me, given that it seems reasonable if it's
waiting for the disk.

To be honest I'd expect shifting 100MB/s over NFS would be pushing one
of your cores pretty hard, so I'm not too surprised your machine is
sluggish with that kind of load.

There are two things which might get some more detail:
   1) Try and capture what happens during a 'hangup; is it a complete hang of the server? Are there any log messages; if you perhaps watch a vmstat 1   during the period of the hangup what's going on (does the IO ever drop to being much lower?)

   2) Try to use something to see what's eating your system time (I'd bet on the rtl ether card perhaps?); the perf   command is good for seeing where system time is going:
       a)  Start your load going
       b) run  sudo perf record -a
            it generates a log file
       c) After 30 seconds or so ctrl-c it
       d) run  sudo perf report --stdio > myperfreport
       e) attach the    myperfreport  to this bugreport so we can see what your kernel is doing.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1071978

Title:
  nfs kernel server is very slow and causing high cpu load

Status in “nfs-utils” package in Ubuntu:
  New

Bug description:
  I have been noticing nfs-kernel server causing very high cpu load and
  the transfers are incredibly slow. In fact for big files, it tends to
  hang up completely and never finish.

  I did try briefly to test the 3.5 backport kernel however that also
  seemed to have the same issue.

  This may be a duplication of bug #879334, however that bug is missing
  log files, and apport-collect told me to file a new bug report
  instead.

  ProblemType: Bug
  DistroRelease: Ubuntu 12.04
  Package: nfs-kernel-server 1:1.2.5-3ubuntu3.1
  ProcVersionSignature: Ubuntu 3.2.0-32.51-generic 3.2.30
  Uname: Linux 3.2.0-32-generic x86_64
  ApportVersion: 2.0.1-0ubuntu14
  Architecture: amd64
  Date: Sat Oct 27 14:50:09 2012
  InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Beta amd64 (20120328)
  ProcEnviron:
   LANGUAGE=en_AU:en
   TERM=xterm
   PATH=(custom, no user)
   LANG=en_AU.UTF-8
   SHELL=/bin/bash
  SourcePackage: nfs-utils
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1071978/+subscriptions