[MERGE] first round of use case benchmarks

[John Arbash Meinel]

Carl Friedrich Bolz wrote:
> John Arbash Meinel wrote:
> [snip]
>> The permutations seem reasonable, though.
>> I wonder if it would be better to separate the timing. 1 time evaluated
>> for generating the bundle, and another time for reading and installing
>> the bundle. Though that ends up doubling the number of tests you write.
> 
> Hum. I agree in principle, but it seems to just take too long to build
> the trees. The test_many_files_big_tree_1000_revision takes a _long_
> time on my machine just as it is, so building the tree twice does not
> sound like it is a good idea to me.

Actually, that is probably a very good idea. We can have separate tests
for up to maybe 100, and then a big 1000 commit test, that just does
both. In the future maybe we can break it up, but in the short term,
that is what the small tests are for.

> 
> Maybe we could add smaller separate writing and installing benchmarks?
> 
> In general it seems that install_bundle is horribly slow, will profile a
> bit.

Which is why we want to benchmark it, and make it faster. Both in the
current form, and in a future bundle format. We may not even want the
really huge tree benchmarks until bundles will actually scale that big.
But as long as the tests are being run unattended, it is probably okay
for it to take a while. (Though 1hr seems a little long)

...

> 
> Attached is an updated version of the patch.
> 
> Cheers,
> 
> Carl Friedrich
> 
> 
> ------------------------------------------------------------------------
> 
> === added file 'bzrlib/benchmarks/bench_bundle.py'
> --- bzrlib/benchmarks/bench_bundle.py	1970-01-01 00:00:00 +0000
> +++ bzrlib/benchmarks/bench_bundle.py	2006-08-30 19:40:51 +0000
> @@ -0,0 +1,218 @@
> +# Copyright (C) 2006 by Canonical Ltd
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as published by
> +# the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
> +
> +"""Tests for bzr bundle performance."""
> +
> +from cStringIO import StringIO
> +import os
> +import shutil
> +
> +from bzrlib import bzrdir
> +from bzrlib.add import smart_add
> +from bzrlib.benchmarks import Benchmark
> +from bzrlib.branch import Branch
> +from bzrlib.bundle import read_bundle
> +from bzrlib.bundle.apply_bundle import install_bundle
> +from bzrlib.bundle.serializer import write_bundle
> +from bzrlib.revision import NULL_REVISION
> +from bzrlib.revisionspec import RevisionSpec
> +from bzrlib.workingtree import WorkingTree
> +
> +
> +class BundleBenchmark(Benchmark):
> +    """Benchmarks for bzr bundle performance and bzr merge with a bundle."""
> +   
> +    def test_create_bundle_known_kernel_like_tree(self):
> +        """Create a bundle for a kernel sized tree with no ignored, unknowns,
> +        or added and one commit.
> +        """ 
> +        self.make_kernel_like_committed_tree()
> +        self.time(self.run_bzr, 'bundle', '--revision', '..-1')
> +
> +    def test_create_bundle_many_commit_tree (self):
> +        """Create a bundle for a tree with many commits but no changes.""" 
> +        self.make_many_commit_tree()
> +        self.time(self.run_bzr, 'bundle', '--revision', '..-1')
> +
> +    def test_create_bundle_heavily_merged_tree(self):
> +        """Create a bundle for a heavily merged tree.""" 
> +        self.make_heavily_merged_tree()
> +        self.time(self.run_bzr, 'bundle', '--revision', '..-1')
> +        
> +    def test_apply_bundle_known_kernel_like_tree(self):
> +        """Create a bundle for a kernel sized tree with no ignored, unknowns,
> +        or added and one commit.
> +        """ 
> +        tree = self.make_kernel_like_tree_committed('tree')
> +
> +        f = open('bundle', 'wb')
> +        try:
> +            write_bundle(tree.branch.repository, tree.last_revision(),
> +                         NULL_REVISION, f)
> +        finally:
> +            f.close()
> +
> +        tree2 = self.make_branch_and_tree('branch_a')
> +        os.chdir('branch_a')
> +        self.time(self.run_bzr, 'merge', '../bundle')

-- you need an extra newline here

> + 
> +class BundleLibraryLevelWriteBenchmark(Benchmark):
> +    """ Benchmarks for the write_bundle library function. """
> +

...

> +    def test_few_files_small_tree_1_revision(self):
> +        os.mkdir("a")
> +        tree, files = self.create_with_commits(5, 1, directory_name="a")
> +        self.commit_some_revisions(tree, files[:5], 1, 1)
> +        self._time_read_write()
> +
> +    def test_few_files_small_tree_500_revision(self):
> +        os.mkdir("a")
> +        tree, files = self.create_with_commits(5, 1, directory_name="a")
> +        self.commit_some_revisions(tree, files[:5], 500, 1)
> +        self._time_read_write()
> +
> +    def test_few_files_small_tree_1000_revision(self):
> +        os.mkdir("a")
> +        tree, files = self.create_with_commits(5, 1, directory_name="a")
> +        self.commit_some_revisions(tree, files[:5], 1000, 1)
> +        self._time_read_write()

It occurs to me that doing 1, 100, 1000 might be a better distribution.
And we might even want to wait on the 1000 for now, if they really are
as bad as they seem. (Or maybe just make an easy way to avoid running
them with the rest of the tests).

...


>      def commit_some_revisions(self, tree, files, num_commits,
> @@ -187,9 +173,11 @@

v- Side note: we should remove this '\' it does nothing but add clutter.

>      testmod_names = [ \
>                     'bzrlib.benchmarks.bench_add',
>                     'bzrlib.benchmarks.bench_bench',
> +                   'bzrlib.benchmarks.bench_bundle',
>                     'bzrlib.benchmarks.bench_cache_utf8',
>                     'bzrlib.benchmarks.bench_checkout',
>                     'bzrlib.benchmarks.bench_commit',
> +                   'bzrlib.benchmarks.bench_info',
>                     'bzrlib.benchmarks.bench_inventory',
>                     'bzrlib.benchmarks.bench_log',
>                     'bzrlib.benchmarks.bench_osutils',
> 

So other than the 1, 100, 1000 split, it looks good enough to me to come in.

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060830/7ef4cd74/attachment.pgp