Optimising branching and merging big repositories between far away locations...

[Asmodehn Shade]

Mmm ok, my mistake it seems to happen on pull -r33 as well... just took a
bit more time...

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18539 autobzr   18   0 1735m 1.7g 2468 S    0 52.6   0:03.11 bzr

I ll rerun with debug output...

Note : I havent changed my repository to format 1.9 yet ( might just hide
the real problem from what I understood... )


2008/10/31 Asmodehn Shade <asmodehn at gmail.com>

> HI again,
>
> I recently got a bzr 1.9rc installed on all of y servers ( some using
> python 2.4.4 , and some python 2.5 )
>
> Bazaar (bzr) 1.9rc1
>   Python interpreter: /usr/bin/python 2.4.4
>   Python standard library: /usr/lib/python2.4
>   bzrlib: /usr/lib/python2.4/site-packages/bzrlib
>   Bazaar configuration: /home/autobzr/.bazaar
>   Bazaar log file: /home/autobzr/.bzr.log
>
> interestingly enough
>
> when I do a :
>
> bzr pull -r33  bzr+ssh://[...]/home/autobzr/deployBZR/Game
>
> the patch you made before about seems to have been applied :
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 31682 autobzr   18   0 82860  77m 2468 S    0  2.4   0:00.25 bzr
>
> however when I do a :
>
> bzr branch -r1 bzr+ssh://[...]/home/autobzr/deployBZR/Game
> \ [=============================                    ] Copying content texts
> 3/5
>
> the patch doesnt seem to apply for some reason :
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  1902 autobzr   24   0 1920m 1.9g 2468 S    0 58.3   0:03.24 bzr
>
> any idea why they might behave differently ?
>
> I didnt run with -Dhpss yet, I can do that if it s needed...
>
> Let me know ;-)
>
> PS I ll come up on IRC now...
>
> --
> Alex
>
> 2008/10/29 John Arbash Meinel <john at arbash-meinel.com>
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Asmodehn Shade wrote:
>> > Thanks a lot for the help John.
>> >
>> > For those who are experiencing the same problem, and as John advised me,
>> > try the sftp:// protocol.
>> > The protocol enforce the size of the chunks being transferred, so that
>> > enable us to workaround the problem as a user, even if it is slower than
>> > bzr+ssh://, at least it works ;-)
>> >
>> > Thanks again !
>> > Waiting for the next version ;-)
>> >
>> > Cheers,
>> >
>> > --
>> > Alex
>> >
>>
>>
>> The other thing we tried was upgrading to --development2 format, to see
>> how much of a difference that would make.
>>
>> To start with, it turns out that the prefetch code doesn't have any
>> effect on this test. When fetching the first revision (which adds
>> thousands of files), we are always making full-width requests.
>>
>> Next, the btree indexes are 23MB instead of 77MB.
>>
>> While he thought he was latency bound, he is actually bandwidth bound
>> while inspecting the indexes. And 'latency bound' later, but because of
>> the buffering time, not because of the actual ping time to the remote
>> host.
>>
>> So it actually was taking 296s before he could start downloading the
>> actual file content. With btree indexes, it starts at around 94s.
>> (77MB/23MB = 3.3, and 296s/94=3.1). And that seems to be a purely
>> bandwidth based issue. His claim was that he has ~1.5Mbit connection,
>> which is approx 180kB/s. 77MB/296s ~ 260kB/s, and 23M/94s ~ 240kB/s.
>>
>> Which means that btree indexes shave about 2.5minutes off of his startup
>> time. Both ways he'll still have to wait a long time to download the
>> 16GB of content (about 19 hours if my numbers are correct).
>>
>> So in the grand scheme of things, it is a small improvement, but it is a
>> consistent improvement. And once he has that enormous initial download,
>> the startup time will become more of an issue.
>>
>> I've also attached an interesting log of my network connection. Both
>> circled areas are doing "bzr log --short -r -10..-1 http://....". Can
>> you guess which one is using btree indexes and which is using knit
>> indexes?
>>
>> The time is 7s versus 15s. Which I have the feeling we are latency bound
>> for all of the .bzr/branch-format, .bzr/branch/format etc files.
>> Otherwise we would see a much more significant change (packing the
>> repository drops off another second or 2, so some of it is having
>> round-trips to multiple pack files, but that still leaves 5s).
>>
>> John
>> =:->
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (Cygwin)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>
>> iEYEARECAAYFAkkIzs0ACgkQJdeBCYSNAAO+8QCeIFCckaJ9MaRqutPBWOob52Kc
>> 6roAn31KdBlgzzHHO8Mpx6oBNxfifWvC
>> =+7+n
>> -----END PGP SIGNATURE-----
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/bazaar/attachments/20081031/f8ad190e/attachment-0001.htm