System Performance under heavy I/O load

[Dr. Nils Jungclaus]

Hi,

I am using 8.04 on several (well equipped) servers and experience the 
following problem on all of them:

When doing larger I/O jobs like backup, I always get a very poor 
interactive response of the system. Interactive in this case means 
performance of database requests, web application requests and even 
interactive tools like top. The usual setup looks like this:

- postgres DB as database backend
- apache as loadbalancer and certificate handler
- several parallel zope instances using zeo
- sometimes more things like vmware-server, samba, postfix

When I start a backup (via network using rsync, local to another HD 
using rsync, or using a USB attached external drive), I get lots of 
delayed processes in top (D), the iowait percentage goes up to 10 to 20 
percent, but the throughput (watched via iostat) is not very high, at 
least far away from the rates I get using only one device. The load goes 
up to 20 or 30, and nothing really gets done by the system. It seams to 
me that the system is standing on it's own feet.

I already tried the following:

- using deadline/cfq scheduler (cfq using ionice for backup processes, 
gives the best results for me, but is still far away from hardware 
capabilities)
- on USB devices, I tried different settings for 
/sys/block/*/device/max_sectors

The hardware is a 24 core Opteron, Adaptec Raid with Raid 10 (getting up 
to 500MB/s read performance) and 64GB Ram
Several other servers (16, 8 Cores, 32/16GB ram, Dell perc6i Raid) 
behave similar.

Are there any hints on getting better I/O performance / better response 
times on such machines?

In my opion, the kernel should be able to schedule the ressources in a 
way that at least any of the hardware components is the restricting 
factor. What I see is a more or less idle system, high load, high iowait 
percentage, no throughput.

Any hints welcome!

    Nils

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-server/attachments/20100421/98e6d4b6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Nils_Jungclaus.vcf
Type: text/x-vcard
Size: 265 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-server/attachments/20100421/98e6d4b6/attachment.vcf>