Optimising branching and merging big repositories between far away locations...

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Asmodehn Shade wrote:
> Hi,
> 
> First thanks for all these details, they are quite interesting and I
> will take some time to investigate a bit ;-)
> 
> 1)
> so I ran  a bzr branch -r1 with -Dphss... and my bzr has been stuck
> there for a few hours now :

...

> 30.376  hpss call w/readv: 'readv',
> '/home/autobzr/deployBZR/.bzr/repository/indices/f65de233ce31cbdace7370611f49937c.tix'
> 30.377                7 bytes in readv request
> 30.609     result:   ('readv',)
> 31.419                65536 body bytes read
> 31.432  hpss call:   'get',
> '/home/autobzr/deployBZR/.bzr/repository/indices/f65de233ce31cbdace7370611f49937c.tix'
> 31.432               (to
> bzr+ssh://deploy.sgf.in.iz/home/autobzr/deployBZR/GameBZR-Live-Temporary/)
> 32.445     result:   ('ok',)
> 60.013                5903449 body bytes read

30s to download 5.9MB is about 200kB/s. If you are on 1.5Mbit, then you
are saturating your pipe with that request.


> 62.310  hpss call w/readv: 'readv',
> '/home/autobzr/deployBZR/.bzr/repository/packs/f65de233ce31cbdace7370611f49937c.pack'
> 62.312                11675 bytes in readv request
> 
> 
> Not sure why it s still stuck like that...
> The progress bar is stuck as well at :
> \ [=============================                    ] Copying content
> texts 3/5
> 
> any possibility to get more logs out of it ?
> 

Well, if you just downloaded 5.9MB of index data, I would guess you have
several hundred megs of content data coming next.

I believe Martin was looking at changing some of the "copy raw bytes"
code to provide progress/indication of how much is being downloaded at
the moment.

> 
> 2) I have bzr 1.8 installed everywhere, but as these repositories are
> big, so I ll give that a try as a last resort ( it might take some time... )

I have code that can generate the indexes directly, rather than doing
the fetch of all data. I'll attach it here in case you are interested.
(When I tested doing an upgrade of a repo with about 600MB it only took
a couple minutes)

> 
> 3) I guess that would be quite efficient since one of my problem is
> latency... however I wish there was an easy way to configure that
> without touching the code ;-) but I ll give it a try to check if I see
> any improvement when I get some time.

Seeing the above, I'm not 100% sure latency is the only issue. I haven't
looked at the whole bzr.log file, but certainly the part you cut out is
bandwidth limited, not latency.

> 
> 4) Last time I tried sftp it was really slower than bzr+ssh ( one the
> same repositories that I am working with right now ), so I ll think I ll
> pass that one...

Sure, every readv is 3 round trips rather than 1. So I would only do it
if it really is the problem. From what you've posted so far, it looks
like something else.


John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkkIf8wACgkQJdeBCYSNAAMCeACfWZjOlFmp5hkIeN6yvD2NE+12
adEAn3FISRuCYWvb4XSosWZsfCcbZhDg
=dwjx
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: convert_to_dev2.py
Url: https://lists.ubuntu.com/archives/bazaar/attachments/20081029/72f5e1e1/attachment.diff