Ubuntu boot speed fall in Hardy

[Krzysztof Lichota]

2008/5/13 Scott James Remnant <scott at canonical.com>:
> On Mon, 2008-05-12 at 23:12 +0200, Krzysztof Lichota wrote:
>
>  > Sorry about that. It just got lost in my mailbox.
>  > And about lack of feedback, I meant feedback from users. The kernel
>  > has been downloaded 280+ times and I got something like 2 reports from
>  > users. I don't know if it works or if it crashes somebody's machines.
>  >
>  Glad to get back in contact!
>
>  I tend to assume that if people aren't complaining, that they're
>  happy ;)  If it was crashing their machines, I'm sure you'd know about
>  it.

Yes, but I also don't know if it is speeding things up. Recently one
person tested this kernel on hardy and reported is slows down boot by
1 second (but speeds up unhibernation).

>  > >  Problem with prefetch is that it's quite a lot of code, in different
>  > >  places, and zero documentation on how it works and which bit does what.
>  >
>  > I agree the documentation needs to be improved. I will add description
>  > of implementation in project wiki. The comparison with other
>  > prefetching solutions would also clear up things a bit.
>  >
>  That would be very useful, the 1000ft overview helps people find their
>  own answers to their questions.

I have already added comparison to readahead in wiki, based on my
reply to the list.

>  > >  For example, how does it determine which blocks need prefetching?
>  >
>  > It monitors page cache to see which pages are used by processes.
>  >
>  What's the "it" in this case?  The kernel module?

Yes, the tracing engine.

>  > >  Where/how are these lists of blocks stored?
>  >
>  > They are stored in /prefetch directory as prefetch lists for each
>  > traced app and for boot stages.
>  > Each file contains list of tuples (device, inode, start-in-pages,
>  > length-in-pages) which describe what to prefetch.
>  >
>  What creates these files?  A userspace daemon or the kernel module
>  itself?  Is this a real filesystem or a virtual one?

The kernel module writes it. For boot prefetching, userspace script
processes the lists as they are merged and sorted for last 3 runs.

It is not a filesystem, it is plain directory on root filesystem. And
files inside are plain files. You can delete them or process them,
like the boot utility does.

>  Shouldn't this be /etc/prefetch? :)

Maybe. Or /boot/prefetch :) It is easy to change.

>  > >  What decides when to load blocks?
>  >
>  > Blocks are loaded when application starts (for application
>  > prefetching) or when appropriate boot script is started (for boot
>  > prefetching).
>  >
>  What determines whether there is a prefetch file for that application?

If the file exists for application, then it is loaded. Prefetch file
names contain part of filename (for humans) and hash of path to
distringuish apps paths.

>  What keeps that up to date?

App prefetching module works like this:
- When application is started, it checks if there is a prefetch file.
If so, it loads it and prefetches files.
- It then starts recording read pages setstimer (by default to 10 seconds).
- When timer runs, it finishes tracing. Next it checks if application
did significant amount of I/O (we don't want to prefetch for little
apps) and if it did, it writes prefetching file for that app.

So prefetch file for application is updated during each application
start. This might change in the future to involve more sophisticated
approach (like averaging over a few runs) to get better results.

>  > >  What if the filesystem isn't mounted yet (/usr), how can the loading be
>  > >  staged?
>  >
>  > Boot prefetching is split into 3 phases: initial boot (with only root
>  > mounted), boot with all partitions mounted and GUI boot. Each stage
>  > has separate prefetching list.
>  >
>  How are these phases delineated?  Does the kernel need to be told what
>  stage it is in, or does userspace determine which set of prefetch files
>  may be used?

Init scripts (similar to readahead scripts) are run and they tell
kernel module which files to load and when.
So boot prefetching can be easily changed by modifying these scripts,
without touching the kernel part.

>  > >  Could we use the lists to sort the LiveCD filesystem generation?
>  >
>  > It depends what you want to do with it. If you want to feed the list
>  > to mksquashfs, it can be done. If you want to add prefetching list to
>  > live CD, this would be harder, as inode numbers are generated during
>  > generation of SquashFS image.
>  >
>  I guess we'd want to order the SquashFS itself, a bit like pre-defrag ;)

Then you can use the trace file, but you have to convert inodes to
paths. This is doable with a simple script.


-- 

	Krzysztof Lichota