[MERGE] Re: knits - slow sftp pushes

[Robert Collins]

On Mon, 2006-05-01 at 10:06 -0500, John Arbash Meinel wrote:
> Robert Collins wrote:
> > we pay versions per file * 2 * round-trip-time on push operations.
> > This occurs because I overlooked it when optimising knits the first  
> > time around. The mechanism is that we write each version as they 
> > stream out of the source knit, which involves two writes: one to the
> > data file and one to the index. One solution is to have the sftp 
> > transport operate in a write-behind fashion and add a 'sync' or 
> > 'flush' style call. I'm not in favour of this. Another approach is
> > to buffer all the writes into a single big append to each of the two
> > files inside the knit join operation. This is safer because it does
> > not involve us writing to the index in advance of data actually 
> > being confirmed by the server.
> > 
> > Thoughts? volunteers to do it?
> > 
> > Rob
> > 
> 
> I think the join() should indeed work by doing buffering. I don't know
> exactly how the code is working, but shouldn't join be doing everything
> in memory, and then flushing to disk when it is done?
> 
> I believe that is how the old weave code worked where you would build up
> your Weave, and then tell it to write to disk. But I didn't follow the
> VersionedFile stuff enough to know if they kept that format.

Well, weave and knit are quite fundamentally different in this regard.

Anyhow, I've done this (and realised I hadn't completely forgotten it -
there was a TODO - I'd just underestimated *how* bad it would be).

also up at http://people.ubuntu.com/~robertc/baz2.0/knit-join-batching/.

Rob
-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: knit-join-batching.patch
Type: text/x-patch
Size: 6742 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060502/ff90dbbf/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060502/ff90dbbf/attachment.pgp