benchmarks under Python 2.5

Martin Pool mbp at canonical.com
Tue Aug 15 08:12:56 BST 2006 

Andrew suggested that we should benchmark bzr under python2.5 before
the final release, so we could report on anything that causes performance
regressions.

I've run that on the same machine (orcadas), using ubuntu's 2.4.3-0ubuntu4 and
2.4.3+2.5b2-0ubuntu1 respectively.  (hm, that second number is a bit
strange...)  The typical output is shown below.

On the things we measure, in this run, Python 2.5 was slightly slower
than 2.4.  I'm not sure yet why that would be; I can think of a few
hypotheses:

 - this package isn't built in the same way, or has more assertions
   turned on in the interpreter

 - some thing that we depend upon has got slower in 2.5 and we shouldn't
   use it

 - python2.5 really is just slower

 - or it's just experimental error

The numbers seem stable within a few seconds overall.

However, I don't think there's anything that justifies dropping our
general work on performance, or asking for a last-minute fix to the
python 2.5 release.

Incidentally these were run on a machine that's being set up as
benchmark.bazaar-vcs.org, and it should soon be running Jan's html
report generator on the results.




with 2.4:
+ ./bzr selftest --benchmark
running tests...
       bzr: /home/mbp/bzr/bench/bzr.dev/bzr
    bzrlib: /home/mbp/bzr/bench/bzr.dev/bzrlib

...s.bench_add.AddBenchmark.test_one_add_kernel_like_tree   OK  2859ms/ 3858ms
...eKernelLikeTreeBenchmark.test_02_make_kernel_like_tree SKIP           0ms
     caching is disabled
...lLikeTreeBenchmark.test_03_make_kernel_like_added_tree   OK  3590ms/ 3605ms
...lLikeTreeBenchmark.test_04_make_kernel_like_added_tree SKIP           0ms
     caching is disabled
...eTreeBenchmark.test_05_make_kernel_like_committed_tree   OK 32401ms/32430ms
...eTreeBenchmark.test_06_make_kernel_like_committed_tree SKIP           0ms
     caching is disabled
...hmark.test_07_make_kernel_like_committed_tree_hardlink SKIP           0ms
     caching is disabled
...MakeKernelLikeTreeBenchmark.test_make_kernel_like_tree   OK  1096ms/ 1097ms
...checkout.CheckoutBenchmark.test_build_kernel_like_tree   OK 19214ms/55022ms
...ch_commit.CommitBenchmark.test_commit_kernel_like_tree   OK 34490ms/38175ms
...t.CommitBenchmark.test_partial_commit_kernel_like_tree   OK  5703ms/ 9518ms
...nch_inventory.InvBenchmark.test_make_10824_inv_entries   OK   263ms/  264ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_cmd_log       OK    66ms/21085ms
...chmarks.bench_log.LogBenchmark.test_cmd_log_subprocess   OK   289ms/21535ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_log           OK  1008ms/22214ms
...b.benchmarks.bench_log.LogBenchmark.test_log_screenful   OK    70ms/21269ms
...chmarks.bench_log.LogBenchmark.test_log_screenful_line   OK    67ms/21210ms
...hmarks.bench_log.LogBenchmark.test_log_screenful_short   OK    35ms/21190ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_log_verbose   OK  1603ms/22749ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_merge_log     OK   273ms/38225ms
...utils.WalkDirsBenchmark.test_walkdirs_kernel_like_tree   OK   116ms/ 1142ms
bzrlib.benchmarks.bench_rocks.RocksBenchmark.test_rocks     OK   222ms/  223ms
...StatusBenchmark.test_no_changes_known_kernel_like_tree   OK  4123ms/36527ms
...atusBenchmark.test_no_ignored_unknown_kernel_like_tree   OK  2176ms/ 5913ms
...ch_transform.TransformBenchmark.test_canonicalize_path   OK   631ms/ 1902ms
...gtree.WorkingTreeBenchmark.test_is_ignored_10824_calls   OK   137ms/  170ms
...gtree.WorkingTreeBenchmark.test_is_ignored_single_call   OK     0ms/   33ms
....WorkingTreeBenchmark.test_list_files_kernel_like_tree   OK   382ms/ 6525ms
...TreeBenchmark.test_list_files_unknown_kernel_like_tree   OK   709ms/40703ms
tests passed
----------------------------------------------------------------------
Ran 29 tests in 426.611s





orcadas% python2.5 ./bzr selftest -v --one --benchmark
       bzr: /home/mbp/bzr/bzr.mbp.python25/bzr
    bzrlib: /home/mbp/bzr/bzr.mbp.python25/bzrlib
running tests...
...ib.benchmarks.bench_add.AddBenchmark.test_one_add_kernel_like_tree   OK  2950ms/ 4021ms
...ch_bench.MakeKernelLikeTreeBenchmark.test_02_make_kernel_like_tree SKIP           0ms
     caching is disabled
...ch.MakeKernelLikeTreeBenchmark.test_03_make_kernel_like_added_tree   OK  3818ms/ 3833ms
...ch.MakeKernelLikeTreeBenchmark.test_04_make_kernel_like_added_tree SKIP           0ms
     caching is disabled
...akeKernelLikeTreeBenchmark.test_05_make_kernel_like_committed_tree   OK 35865ms/35895ms
...akeKernelLikeTreeBenchmark.test_06_make_kernel_like_committed_tree SKIP           0ms
     caching is disabled
...LikeTreeBenchmark.test_07_make_kernel_like_committed_tree_hardlink SKIP           0ms
     caching is disabled
...bench_bench.MakeKernelLikeTreeBenchmark.test_make_kernel_like_tree   OK  1119ms/ 1121ms
...marks.bench_checkout.CheckoutBenchmark.test_build_kernel_like_tree   OK 20668ms/55741ms
...nchmarks.bench_commit.CommitBenchmark.test_commit_kernel_like_tree   OK 32799ms/36693ms
....bench_commit.CommitBenchmark.test_partial_commit_kernel_like_tree   OK  6136ms/10008ms
...enchmarks.bench_inventory.InvBenchmark.test_make_10824_inv_entries   OK   282ms/  283ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_cmd_log                   OK    64ms/22104ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_cmd_log_subprocess        OK   270ms/22492ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_log                       OK  1009ms/23060ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_log_screenful             OK    72ms/22084ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_log_screenful_line        OK    65ms/22116ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_log_screenful_short       OK    34ms/22097ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_log_verbose               OK  1637ms/23687ms
bzrlib.benchmarks.bench_log.LogBenchmark.test_merge_log                 OK   272ms/38659ms
...rks.bench_osutils.WalkDirsBenchmark.test_walkdirs_kernel_like_tree   OK   136ms/ 1237ms
bzrlib.benchmarks.bench_rocks.RocksBenchmark.test_rocks                 OK   204ms/  205ms
...ench_status.StatusBenchmark.test_no_changes_known_kernel_like_tree   OK 13414ms/47251ms
...ch_status.StatusBenchmark.test_no_ignored_unknown_kernel_like_tree   OK  2019ms/ 5932ms
...nchmarks.bench_transform.TransformBenchmark.test_canonicalize_path   OK   633ms/ 2064ms
...bench_workingtree.WorkingTreeBenchmark.test_is_ignored_10824_calls   OK   130ms/  169ms
...bench_workingtree.WorkingTreeBenchmark.test_is_ignored_single_call   OK     0ms/   37ms
..._workingtree.WorkingTreeBenchmark.test_list_files_kernel_like_tree   OK   381ms/ 4484ms
...tree.WorkingTreeBenchmark.test_list_files_unknown_kernel_like_tree   OK   612ms/42208ms

----------------------------------------------------------------------
Ran 29 tests in 447.515s

OK
tests passed



-- 
Martin

More information about the bazaar mailing list