System has 10%+ iowait

[James Andrewartha]

I've got a recently installed hoary (5.04) system that spends a lot of its 
time in iowait. The specs are: HP Proliant DL 360, dual 3.4GHz xeon with HT, 
1GB ram, two Seagate 7200.7 160GB SATA drives in RAID 1. It's running in 
EM64T mode, and is a general web/mail/dns/spam/lists/gateway server. The 
workload of this server was adequately handled by the dual p3 733 it 
replaced, so it's not like I'm thrashing it.

Basically, anything that touches the disk ends up in the D state for several 
seconds, including things like kjournald, syslog and postmaster (for 
postgres, for dspam). I've tried the default 2.6.10 kernel, and 2.6.11 from 
universe, and booting 2.6.10 with irqpoll, and breaking the RAID, but none 
of this has an effect. The only thing I can thing of is it's using ata_piix 
for the "Intel Corp. 6300ESB SATA" controller, and that using ahci.ko 
instead might fix it, but I really don't know.

Some stats:
vmstat 1:
  0  2   4952  53992  25948 546584    0    0     0  1404 1377   120  0  1 72 28
  0  1   4952  54024  25948 546584    0    0    44   220 1382   171  0  1 70 30
  0  3   4952  53896  25960 546640    0    0    80  4900 1317   111  1  1 63 35
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
  2  3   4952  53896  25964 546636    0    0     4  1432 1419    75  0  1 37 63
  0  2   4952  53896  25972 546696    0    0     0   740 1415    64  0  1 33 67
  0  1   4952  54112  25980 546688    0    0    16   508 1311   178  1  1 74 25
  0  0   4952  55728  26132 547012    0    0   380   456 1178   373  0  1 83 16
  0  0   4952  56312  26132 547012    0    0     0     0 1062    53  0  0 100  0
  0  0   4952  56312  26132 547012    0    0     4     0 1061    63  0  0 100  1

And a graph of cpu usage (user,system,nice) and iowait:
http://quoll.daa.com.au/stats/cpuusage.png

Thanks for any suggestions,

-- 
James Andrewartha
Systems Administrator
Data Analysis Australia Pty Ltd