load average 34.73 !

[john]

Hi all,

I am running 4 cores with 16 G ram.  I have a single 1G nic serving 45 stations.
Right now I have 38 users on and my load average is 34.73
I've seen it spike to 70. I've pasted an excerpt from TOP below.


top - 08:44:10 up 1 day, 14:56, 38 users,  load average: 34.73, 34.76, 21.83
Tasks: 1438 total,  10 running, 1426 sleeping,   0 stopped,   2 zombie
Cpu(s): 64.8%us, 21.3%sy,  4.6%ni,  0.8%id,  6.1%wa,  0.1%hi,  2.4%si,  0.0%st
Mem:  16632608k total,  9663840k used,  6968768k free,   406064k buffers
Swap:  9936160k total,        0k used,  9936160k free,  3951436k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13847 auer_saw  20   0  228m  98m  27m R   45  0.6   1:05.38 firefox
21212 murphy_c  20   0  273m 110m  26m S   29  0.7   0:39.00 firefox
 5203 clements  20   0  159m  59m  21m D   27  0.4   0:14.52 firefox
 4883 hembree_  20   0  373m 181m  28m R   26  1.1   2:05.64 firefox
27114 parks.ta  20   0  248m 106m  25m S   18  0.7   1:53.48 firefox
24714 smith_ni  35  15 35836  19m  12m R   17  0.1   1494:26
gnome-panel
19552 straube.  20   0  162m  61m  22m S   13  0.4   0:04.91 firefox
 3344 weber_my  20   0  266m 108m  25m S   10  0.7   1:23.35 firefox
13935 challman  20   0  280m 130m  26m S   10  0.8   2:40.12 firefox

I ran vmstat and I see that I have lots of interrupts and context
switches, and also processes waiting for attention.

root at vbuntu:/var/log# vmstat 5 200
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  1      0 9555716 420520 4682252    0    0     7    28   73   47 18  1 80  1

10  1      0 9550352 420572 4676348    0    0   562  3374 4284 17322 45 21 25  9
 1  1      0 9595204 420600 4683272    0    0  1221  1410 3730 6236 45 14 30 11
 1  0      0 9655864 420624 4677924    0    0   550  1334 4091 8094 44  9 39  8
 2  2      0 9654436 420628 4677756    0    0     0  2533 3836 6361 30  5 60  5
 4  0      0 9743600 420652 4677408    0    0     0   166 3663 5912 33  6 61  0
 1  1      0 9816940 420672 4681848    0    0   439   202 3421 5607 35  7 49  9
 3  1      0 9775504 420708 4693380    0    0  1000  1069 3771 4855 39  6 38 16
 4  0      0 9742368 420716 4688016    0    0   105  2130 3539 4844 36  5 55  4
 4  0      0 9754888 420716 4688128    0    0     0   222 3389 5058 32  4 64  0
 1  0      0 9756060 420720 4688072    0    0     0    82 2947 3813 26  4 70  0
 1  0      0 9881396 420736 4688160    0    0     0   218 3683 6299 36  8 57  0
 1  0      0 9881144 420744 4688208    0    0     0   124 3006 3872 28  3 69  0
 1  2      0 9865324 420756 4699228    0    0   198  2128 3130 3660 36  5 50  9
 1  0      0 9871528 420776 4690000    0    0   134  2975 3532 3999 39  7 47  7
 4  0      0 9874736 420800 4691984    0    0    22   142 3323 3826 30  3 66  0
 2  0      0 9842856 420828 4706212    0    0   162  3079 3330 3998 51  6 37  6
 4  0      0 9830752 420872 4710852    0    0    58  1092 3156 3737 54  4 41  1
 1  0      0 9846836 420876 4693096    0    0    38   388 3471 5125 43  4 52  0
 1  0      0 9847376 420876 4693104    0    0     0  3789 3805 5539 30  5 60

As you can imagine the user experience right now it not great. Can
anyone give me some ideas
about reducing load on the system?

Thanks!

John