Load average very high 200-300 ,CPU 99% idle,how to check what is causing high load average Ubuntu 14.04.3

[mad Engineer]

I am running Ubuntu 14.04 3.13.0-61-generic kernel on  servers with 40
cores and 250 Gb RAM.But for some strange reasons load average is showing
unacceptable values of 200-300 but CPU is 99% idle.Even though there is no
impact on application,i like to know why is it showing these values.

*observation:*

*top -d 1*

top - 04:44:41 up 10 days,  2:27,  2 users,  load average: 203.71, 211.25,
217.05

Tasks: 1062 total,   1 running, 1061 sleeping,   0 stopped,   0 zombie

%Cpu(s):  0.1 us,  0.2 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,
0.0 st
KiB Mem:  26410931+total, 84802192 used, 17930710+free,   204520 buffers


*cat /proc/loadavg*

214.55 213.37 217.71 1/1094 41477


*dstat -n*

-net/total-

recv  send

   0     0

267k   10k

217k   27k

289k  254k


*mpstat 1 1*

Linux 3.13.0-61-generic (san3r29o011.san3.cci.att.com)  11/14/2015
_x86_64_        (40 CPU)



05:01:29 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft   %steal
%guest  %gnice   %idle

Average:          all       0.65    0.00       0.80     0.00        0.00
0.00     0.00     0.00        0.00      98.55


Any idea why server is behaving strangely??


I have tried different forums and IRC rooms none could explain the
behavior.Can some body please have a look at this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-users/attachments/20151114/ec8b812a/attachment.html>