[Bug 336652] [Fwd: Re: [Fwd: I/O performance regression]]

[Scott James Remnant]

-------- Forwarded Message --------
From: Andy Whitcroft <apw at canonical.com>
To: Scott James Remnant <scott.james.remnant at canonical.com>
Cc: Tim Gardner <tim.gardner at canonical.com>, Pete Graner
<pgraner at canonical.com>
Subject: Re: [Fwd: I/O performance regression]
Date: Mon, 2 Mar 2009 11:37:24 +0000

On Sat, Feb 28, 2009 at 01:06:26PM -0500, Pete Graner wrote:

> Scott was making reference to seeing poor I/O performance, and this being a
> limiting factor for boot time.  He pointed to this bug:
> 
> http://bugzilla.kernel.org/show_bug.cgi?id=12309

Well the first thing to note about this bug is that it is not a report
of poor IO performance, quite the opposite, that IO is perfectly good but
interactivity goes to hell as IO fairness goes out the window.  I suppose
if the issue is that new programs and shells are delayed as a result that
might impact boot performance if we have a background readahead in play.

During some experimentation I did notice that one of my build boxes has
its io scheduler set to deadline?  Not something I have done deliberatly,
and differing from an almost identicle build box made and upgraded from
the same media in lockstep:

    apw at lana$ dmesg | grep io\ scheduler
    [    1.253192] io scheduler noop registered
    [    1.253193] io scheduler anticipatory registered
    [    1.253195] io scheduler deadline registered (default)
    [    1.253248] io scheduler cfq registered
    apw at lana$ cat /sys/block/sda/queue/scheduler
    noop anticipatory [deadline] cfq 

I have tried the suggested trigger in the bug running a big dd (which
is a write version of readahead case) and that cirtainly buggers things
up badly if you are using deadline, it is almost impossible to create
new windows or switch to an alternative VT.  I tried using ionice in
combination to little effect.  Switching to cfq improved things hugely.
We should check the default that has been picked up on your test rig.

There is much conjecture in the bug you link about bugs in both the IO
schedulers and the cpu scheduling too.  It has been suggested that
changing time source can help.  So it might also we worth switching
timesource for comparison.

    apw at dm$ cat /sys/devices/system/clocksource/clocksource0/* 
    hpet acpi_pm jiffies tsc 
    hpet

In theory at least you should be able to boot with clocksource=<source>
for each of the ones listed on the first line there, and confirm it is
enabled.

It would also be interesting to see how adjusting the queue depth for
the disk affects these delays, try the test with the queue at 128 (the
default) 64, 32, and 16 and see how that affects the latencies you see:

    echo N > /sys/block/sda/queue/nr_requests

It is entirly possible we would need different parameters for the boot
phase to typicial desktop usage.

Finally which kernel is this testing occuring on.

-apw
-- 
Scott James Remnant
scott at canonical.com

-- 
Poor system performance under I/O load
https://bugs.launchpad.net/bugs/336652
You received this bug notification because you are a member of Kernel
Bugs, which is subscribed to Linux.