psusi at cfl.rr.com
Sun Dec 11 02:40:19 GMT 2005
David Nielsen wrote:
> As a sidenote to this the upcoming Reiser4 filesystem is able to use a
> repacker (same thing as defragmentation tool really), so I wouldn't be
> surprised if we would have this functionality in the near future.
> Historically our filesystems have laid out data in a near optimal way so
> defraging hasn't been a problem untill now, and it always posed a
> problem being a slow cumbersome process - I'm my impression that the
> Reiser4 repacker will run little by little in the background using only
> idle time to repack the disk so this wouldn't even be felt by the user
> as compared to the full defrag tools someone like Norton would provide
> for Windows.
I actually wrote the maintainer of libreiser to ask if he thought it
would be possible to write a utility using it to do that kind of
repacking. His reply was no, libreiser does not provide the required
functionality, but libreiser4 does. Unfortunately, reiser4 is caught up
in politics at the moment and it doesn't look like it will make it into
Tristan Wibberley wrote:
> Or it could be single-threaded, so a read is issued, then completes,
> such that the head is in the wrong place for the next read, which is
> subsequently issued, leaving the head in the wrong place for the next,
> etc. If so, then using asynchronous mechanisms could provide a
> performance boost (the speed I'm getting for readahead-list seems far to
> small for the normal level of fragmentation I expect to have). If it
> already *is* asynchronous then playing with the kernel tunables before
> running readahead and resetting them afterwards could help by extending
> the I/O queue length and reducing the fairness to allow the reads to be
> ordered better.
I have been a big proponent of async IO since 1998, when I wrote an ftp
server on NT using zero copy overlapped IO and IO completion ports and
was blown away by network throughput I never dreamed was possible using
virtually zero cpu time. I actually modified dd the other day to use
linux native async IO in conjection with O_DIRECT and the results were
very positive. In this case though, it is not really appropriate.
Async IO's main advantage is that you can carry out a very large amount
of IO with many different targets with very little cpu overhead.
Because of readahead mechanisms, conventional blocking IO is capable of
basically just as much throughput for sequantial IO, even if it costs a
good deal more cpu cycles. In the case of readahead-list, it does not
directly read but instead asks the kernel to precache the file with the
readahead() system call, which it does quite efficiently.
This is why I am reasonably sure the problem is simply disk seeking
between each file as they are scattered randomly around the disk, and a
number of them are actually rather small.
> I don't have more experience (none whatsoever), but that looks really
> interesting. Could you send me details of what you did, and any diffs
> against stock scripts? I'd like to see this and have a play.
I actually didn't even modify any scripts. I just asked grub to pass
the break=init parameter to the kernel to have the initramfs drop me to
a shell just before passing to the real root filesystem. I then mounted
the root filesystem in /root, loaded the loop, cramfs, and unionfs
modules, configured loop0 to connect to /root/readahead.cramfs and
mounted loop0 in /readahead. Then I did a "mount -t unionfs -o
dirs=/readahead=ro:/root=rw none /real-root" to mount the root
filesystem with the readahead cramfs overlaid on top in /real-root.
Then I executed run-init with /real-root in place of /root, and then
there were a bunch of decompression errors I think from cramfs, then the
kernel paniced because init terminated. I think init was loading some
libraries in the readahead cramfs.
> The biggest problems I see during boot (before gdmgreeter goes quiet)
> are modprobe in the initramfs and elsewhere, hwclock, hald-probe-volu
> (whatever that is short for), Xorg and gdmgreeter startup, or the fact
> that things wait for others where they shouldn't need to. In most cases
> they *do* need to but ought not - that requires some big surgery to
> packages to fix it properly which I guess is certainly *not* going to
> happen for dapper.
Yea, I also noticed the modprobe and things like X waiting for all the
hardware detection to finish before starting. I guess it needs the
mouse and whatnot to be detected before it can start, but it would be
nice if it could get started sooner, then connect to the mouse once it
is detected. I don't even see gdmgreeter listed in my bootchart at all
More information about the ubuntu-devel