Bazaar 1.6 released - some benchmarks.

[David Ingamells]

I have to say that my first experience with bzr 1.6 performance is very 
disappointing. I observe times more that 300% slower than bzr version 
1.5 (which itself showed no improvements over bzr 1.2). Checkout 
--lightweight is, in some cases, even worse.

Working with a repos with only a few revisions (33) but LOTS of files:

cmsadmin at mscvs01:/home/CmsRoot/repos/BSW/12.1$ bzr info
Repository branch (format: pack-0.92)

I timed a bzr+ssh:// branch with versions 1.2, 1.5 and 1.6. Both source 
and destination disks were local to their machines.

bzr_1.2:
=====
david.ingamells at msdes002: time bzr branch 
bzr+ssh://mscvs01/home/CmsRoot/repos/BSW/12.1 branch_1.2
Branched 33 
revision(s).                                                                                                                                                          


real    2m45.985s
user    1m14.725s
sys     0m55.023s

bzr 1.5:
=====
david.ingamells at msdes002: time /data/id/bin/bzr branch 
bzr+ssh://mscvs01/home/CmsRoot/repos/BSW/12.1 branch_1.5
Branched 33 
revision(s).                                                                                                                                                          


real    2m48.375s
user    1m13.581s
sys     0m56.372s


bzr 1.6:
=====
david.ingamells at msdes002: time bzr branch 
bzr+ssh://mscvs01/home/CmsRoot/repos/BSW/12.1 branch_1.6
Branched 33 
revision(s).                                                                                                                                                          


real    10m31.409s
user    5m2.215s
sys     5m16.720s

bzr 1.6 stacked
==========
david.ingamells at msdes002: time bzr branch --stacked 
bzr+ssh://mscvs01/home/CmsRoot/repos/BSW/12.1 branch_1.6_stack
using <bzrlib.repofmt.pack_repo.RepositoryFormatKnitPack5 object at 
0x876decc> for stacking
Created new stacked branch referring to 
bzr+ssh://mscvs01/home/CmsRoot/repos/BSW/12.1/.                                                                                           


real    17m21.220s
user    8m12.383s
sys     8m56.226s

Similar timings with the file:// protocol and checkout --lightweight gave:

pack-0.92
                   branch bzr+ssh://   branch file://   checkout 
--lightweight bzr+ssh://
                   =================   ==============   
=================================
bzr 1.2            2m45.985s           1m6.795s         3m0.301s
bzr 1.5            2m48.375s           1m3.309s         3m7.716s
bzr 1.6            10m31.409s          1m33.950s        17m44.029s
bzr 1.6 stacked    17m21.220s          1m20.099s              

With both protocols bzr 1.6 is for my repositories a retrograde release 
as far as branch performance is concerned.

I repeated the same tests with another repository having lots of history 
(3644 revisions) but fewer files and with dirstate-tags format:

dirstate-tags
                   branch bzr+ssh://   branch file://   checkout 
--lightweight bzr+ssh://
                   =================   ==============   
=================================
bzr 1.2            1m0.376s            1m1.186s         0m37.084s
bzr 1.5            1m52.103s           1m58.407s        0m37.400s
bzr 1.6            4m8.584s            2m19.070s        1m4.605s
bzr 1.6 stacked    1m11.704s           1m2.370s        

The branch sizes are:

$ du -sk *
513384  branch_1.2
513380  branch_1.5
513380  branch_1.6
296960  branch_1.6_stack
285328  toto_1.2
285324  toto_1.5
244948  toto_1.6
97140   toto_1.6-stack


Given the performance of bzr 1.6 with pack-0.92 I feel very cynical 
about the advice produced when I ran the bzr 1.6 branch command on the 
dirstate-tags repos - it was so important it had to be repeated! (with 
the --stacked option there were 5 messages such produced).

Format <RepositoryFormatKnit1> for 
chroot-140730668:///home/CmsRoot/repos/CVTOOL/.bzr/ is deprecated - 
please use 'bzr upgrade' to get better performance
Format <RepositoryFormatKnit1> for 
bzr+ssh://mscvs01/home/CmsRoot/repos/CVTOOL/.bzr/ is deprecated - please 
use 'bzr upgrade' to get better performance

 From my measurements "better performance"is only notable by its 
absence. Admittedly it looks like 1.6 branch --stacked will be faster 
provided there is lots of history in the repos that can be left behind. 
Also --stacked with such a repos saves lots of disk space.

However the bottom line is that version 1.6's branch with pack-0.92 and 
bzr+ssh:// protocol is about 375% slower than in version 1.5 and 382% 
slower than version 1.2. With file:// protocol the figure is 221% 
compared to bzr 1.5.

I ran these tests with lots of hope for improvements, and really wanted 
to be able to send good news. I would love it if someone can point out a 
stupid error I've made which will turn the figures around. Maybe I've 
misunderstood the intentions of stacked branches. I have shown here that 
they do save lots of disk space, but I had also expected performance 
similar to bzr 1.5's checkout --lightweight. I have shown that branch 
--stacked and checkout --lightweight are similar in version 1.6 (which 
makes sense), but why are all the timings much slower than their 
equivalents in bzr 1.5???



John Arbash Meinel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Finally, the long awaited bzr 1.6 has been released.
>
>