Optimising branching and merging big repositories between far away locations...

[Asmodehn Shade]

HI again,

I recently got a bzr 1.9rc installed on all of y servers ( some using python
2.4.4 , and some python 2.5 )

Bazaar (bzr) 1.9rc1
  Python interpreter: /usr/bin/python 2.4.4
  Python standard library: /usr/lib/python2.4
  bzrlib: /usr/lib/python2.4/site-packages/bzrlib
  Bazaar configuration: /home/autobzr/.bazaar
  Bazaar log file: /home/autobzr/.bzr.log

interestingly enough

when I do a :

bzr pull -r33  bzr+ssh://[...]/home/autobzr/deployBZR/Game

the patch you made before about seems to have been applied :
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31682 autobzr   18   0 82860  77m 2468 S    0  2.4   0:00.25 bzr

however when I do a :

bzr branch -r1 bzr+ssh://[...]/home/autobzr/deployBZR/Game
\ [=============================                    ] Copying content texts
3/5

the patch doesnt seem to apply for some reason :
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1902 autobzr   24   0 1920m 1.9g 2468 S    0 58.3   0:03.24 bzr

any idea why they might behave differently ?

I didnt run with -Dhpss yet, I can do that if it s needed...

Let me know ;-)

PS I ll come up on IRC now...

--
Alex

2008/10/29 John Arbash Meinel <john at arbash-meinel.com>

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Asmodehn Shade wrote:
> > Thanks a lot for the help John.
> >
> > For those who are experiencing the same problem, and as John advised me,
> > try the sftp:// protocol.
> > The protocol enforce the size of the chunks being transferred, so that
> > enable us to workaround the problem as a user, even if it is slower than
> > bzr+ssh://, at least it works ;-)
> >
> > Thanks again !
> > Waiting for the next version ;-)
> >
> > Cheers,
> >
> > --
> > Alex
> >
>
>
> The other thing we tried was upgrading to --development2 format, to see
> how much of a difference that would make.
>
> To start with, it turns out that the prefetch code doesn't have any
> effect on this test. When fetching the first revision (which adds
> thousands of files), we are always making full-width requests.
>
> Next, the btree indexes are 23MB instead of 77MB.
>
> While he thought he was latency bound, he is actually bandwidth bound
> while inspecting the indexes. And 'latency bound' later, but because of
> the buffering time, not because of the actual ping time to the remote host.
>
> So it actually was taking 296s before he could start downloading the
> actual file content. With btree indexes, it starts at around 94s.
> (77MB/23MB = 3.3, and 296s/94=3.1). And that seems to be a purely
> bandwidth based issue. His claim was that he has ~1.5Mbit connection,
> which is approx 180kB/s. 77MB/296s ~ 260kB/s, and 23M/94s ~ 240kB/s.
>
> Which means that btree indexes shave about 2.5minutes off of his startup
> time. Both ways he'll still have to wait a long time to download the
> 16GB of content (about 19 hours if my numbers are correct).
>
> So in the grand scheme of things, it is a small improvement, but it is a
> consistent improvement. And once he has that enormous initial download,
> the startup time will become more of an issue.
>
> I've also attached an interesting log of my network connection. Both
> circled areas are doing "bzr log --short -r -10..-1 http://....". Can
> you guess which one is using btree indexes and which is using knit indexes?
>
> The time is 7s versus 15s. Which I have the feeling we are latency bound
> for all of the .bzr/branch-format, .bzr/branch/format etc files.
> Otherwise we would see a much more significant change (packing the
> repository drops off another second or 2, so some of it is having
> round-trips to multiple pack files, but that still leaves 5s).
>
> John
> =:->
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (Cygwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkkIzs0ACgkQJdeBCYSNAAO+8QCeIFCckaJ9MaRqutPBWOob52Kc
> 6roAn31KdBlgzzHHO8Mpx6oBNxfifWvC
> =+7+n
> -----END PGP SIGNATURE-----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/bazaar/attachments/20081031/c57a7d31/attachment.htm