SCM discussions in the GNOME community

Fri Sep 21 19:14:09 BST 2007

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hendy Irawan wrote:
> Andrew Cowie wrote:
>> On Tue, 2007-09-11 at 18:16 +0700, Hendy Irawan wrote:
>>
>>  
>>> I've been having about Bazaar performance locally. But anytime I want
>>> to "bzr push sftp://" it's NIGHTMARE.
>>>     
>>
>> Yeah. The round trip cost of twice per file is devastating when working
>> transcontinental with one of the dumb protocols (http://, sftp://) -
>> both on push and checkout, pull.
>>
>> So, don't use a dumb server.

Right. Actually 'bzr push sftp://" is even worse than that. Because of
certain restrictions it is 6 round trips per versioned file.
Specifically to do an append you have to OPEN, STAT, WRITE, (CLOSE).
(sftp doesn't have an append command, so you have to stat and ask it to
write at the end of the file, I consider CLOSE to be extra because I
believe it is non-blocking).

And the way knits are written we have 2 files (data file, index file) so
you get 3x2 = 6 round trips per versioned file.

Now, after the first push, the rest are a bit better, because it is only
6x per changed file. Still not good, but at least you don't pay the full
tree expense every time.

Further, Launchpad now supports "bzr push bzr+ssh://". Even if you have
an older client, 'bzr+ssh://' has an "APPEND" function, so it can do a
single round trip rather than 3.

I've been away, so I haven't seen if Andrew's streaming work has been
merged (as near as I can tell it has not, which makes me sad :(

Anyway, when that is finally merged, then bzr+ssh will be able to stream
all changes in a single large stream, rather than paying a round trip
each time. I think he showed something like a 7x performance improvement
by doing so.

Also, when Robert's Packs become the default format (I think they will
be available but experimental in 0.92), then we will have
~log_10(Num_revisions) files for data storage, rather than
2xNum_versioned_files. He still has some issues for incremental commits
and how the indexes are updated. But it might make sftp reasonable again.

>>
>> Since you've got ssh access to the target server, hopefully you can
>> install Bazaar there. If you can, switch to pushing by using bzr+ssh://.
>> This will run the "smart server" on the server side when you push and
>> use the newer protocols over the wire. It's been improving steadily over
>> the last 3-4 releases and is enormously better than sftp://.
>>
>> There is likewise something to be said for running their server program
>> as a daemon server-side and publishing bzr:// as the URL to your
>> branches [you can, in theory, arrange http:// to do this transparently
>> if you've got good mod_python foo, although I haven't succeeded at this
>> yet myself]
>>
>>   
> Ah... Great suggestion :)
> 
> So, I'd push to my server first, then Launchpad will mirror it...... :)
> I think that will work...... for now.
> 
> I'm actually not very fond of hosting this in my own server. It's a
> (supposed to be) community project anyway.
> 
> I wonder why Launchpad doesn't provide a smart server as default...... :-(

It does.

> 
> At least now I know a workaround. I'll try to cope with it now, and if
> someday I notice it's getting slow as hell, I can switch to the more
> "complicated", smart server solution. That way I can sleep well. :-)
> 
> Hopefully at that time, Bazaar will already have a better algorithm for
> dealing with dumb servers. :)

We will.

> 
>>> What I think is that the PR is somewhat misguided
>>>     
>>
>> Make sure you use the latest (ie, >= 0.90) version of Bazaar; otherwise
>> the whole conversation is moot. The developers have been making huge
>> strides forward and there is little point in not taking advantage of it.
>> [ie, forget the usual Debian-esque tendency to think that somehow a year
>> old release is somehow more stable than the latest one]
>>   
> Yep I do use 0.90. Also with bzr-svn 0.42, but I guess that one isn't
> relevant.
> 
> A note is that bzr-svn imports my Subversion repositories really fast,
> And when it works it just works(tm).
> 
> SuperKudos to Jelmer Vernooij for that one :)
>> Finally (and admittedly this is a hard sell from the standpoint of the
>> massive PR problem Bazaar has), but a lot of their earlier work was
>> focused on performance and correctness during operations like merging.
>> This isn't the first impression people get of Bazaar, unfortunately, but
>> it makes a big difference once you're underway and managing a growing
>> project. It works, it works right, and it works fast.
>>   
> OH YES INDEED!!
> 
> I've bitten by SVK a few times by this actually........ When a merge got
> wrong I can pissed as hell. And even more if it didn't warn me about it
> at all.
> 
> SVK had troubles with me regarding line-endings in Windows-Linux. It
> probably because I didn't set the proper line endings SVN properties.
> But seriously, in nowadays you can't ever be sure which OS you're gonna
> be using, or the people who contribute, so the tools needs be a little
> wee bit smarter about it.
> 
> I'm not sure how Bazaar handles line-endings now, when working with
> people in different operating systems.. and will happen if someone tries
> to merge a file when the line endings are different.... Hopefully you
> guys have figured this out long ago...
>> AfC
>> Sydney
>>   
> After scouring the net, I once thought this was "Agenda for Change" to
> "Automatic Frequency Control" to... Andrew Frederick Cowie.
> I guess the last one is the real one. =))
> 

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG9AnxJdeBCYSNAAMRAiekAJ9Vsw7/EN4uuQRt6/KJ++aYR9GQpACgkiSu
kMhb40x7I5snfjn3N7AEpdE=
=5H6A
-----END PGP SIGNATURE-----