[MERGE] Benchmark caching intermediate trees

[John Arbash Meinel]

The attached patch implements the ability for the benchmark suite to
cache an intermediate tree, which can then be copied or hardlinked to
run other tests, rather than having to regenerate it from scratch all
the time. The user can specify a cache directory to use, which can be
re-used between test runs. If they don't specify one, then a directory
underneath the TEST_ROOT is used, so it will be automatically cleaned up.

In my testing, it can make a huge difference when running more than one
benchmark test. Specifically:

1) making a kernel like tree takes 5-8s. hardlinking that to a new tree
takes 1s.
2) adding all files to a kernel-like tree takes another 5-10s. Copying
that to a new tree, including cloning the working tree takes only 1-1.5s.
3) committing a freshly added tree takes 60-80s. hardlinking the working
tree and copying the .bzr files takes 20s, and hardlinking both the
working tree and .bzr files takes 7s.

The biggest issue is that all of this time is cumulative. So without any
caching, creating a new kernel-like tree with one commit takes as much
as 70+10+8=88s just to test how long it takes to run 'bzr status'.
With caching, you can do the setup in only 7 seconds. (At least for the
second test).

These changes really make it a lot easier to run new benchmarks, because
we don't spend as much time re-creating the same thing over and over again.

I also mark all of the hardlinked files as readonly, rather than leaving
them readwrite. So that should help prevent tests from stepping on
eachother's toes.

These are the raw numbers:

1) Just create a kernel like tree
...lLikeTreeBenchmark.test_01_make_kernel_like_tree   OK  6868ms/ 6919ms

2) hardlink the working files into another location
...lLikeTreeBenchmark.test_02_make_kernel_like_tree   OK   859ms/10002ms

3) hardlink the working files, and smart_add() them all to a new branch
...reeBenchmark.test_03_make_kernel_like_added_tree   OK  6676ms/ 6689ms

4) hardlink the working files, and copy the (3) branch
...reeBenchmark.test_04_make_kernel_like_added_tree   OK  1241ms/ 3940ms

5) hardlink the working the tree from (4), and copy the branch, and then
run 'tree.commit()' on the result
...enchmark.test_05_make_kernel_like_committed_tree   OK 70304ms/70330ms

6) Hard-link the working tree from (5), and copy the .bzr directory
...enchmark.test_06_make_kernel_like_committed_tree   OK 14839ms/30925ms

7) hard-link the working tree and .bzr from (5)
...test_07_make_kernel_like_committed_tree_hardlink   OK  7156ms/15361ms

I think dropping the setup time from almost 90ms down to
7-15s for each test that wants to use a real kernel tree seems
worthwhile to me.

John
=:->
-------------- next part --------------
A non-text attachment was scrubbed...
Name: benchmark-caching.patch
Type: text/x-patch
Size: 69825 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060807/b25d9d67/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060807/b25d9d67/attachment.pgp