[MERGE] Re: knits - slow sftp pushes

[John Arbash Meinel]

Robert Collins wrote:
> On Mon, 2006-05-01 at 10:06 -0500, John Arbash Meinel wrote:
>> Robert Collins wrote:
>>> we pay versions per file * 2 * round-trip-time on push operations.
>>> This occurs because I overlooked it when optimising knits the first  
>>> time around. The mechanism is that we write each version as they 
>>> stream out of the source knit, which involves two writes: one to the
>>> data file and one to the index. One solution is to have the sftp 
>>> transport operate in a write-behind fashion and add a 'sync' or 
>>> 'flush' style call. I'm not in favour of this. Another approach is
>>> to buffer all the writes into a single big append to each of the two
>>> files inside the knit join operation. This is safer because it does
>>> not involve us writing to the index in advance of data actually 
>>> being confirmed by the server.
>>>
>>> Thoughts? volunteers to do it?
>>>
>>> Rob
>>>
>> I think the join() should indeed work by doing buffering. I don't know
>> exactly how the code is working, but shouldn't join be doing everything
>> in memory, and then flushing to disk when it is done?
>>
>> I believe that is how the old weave code worked where you would build up
>> your Weave, and then tell it to write to disk. But I didn't follow the
>> VersionedFile stuff enough to know if they kept that format.
> 
> Well, weave and knit are quite fundamentally different in this regard.
> 
> Anyhow, I've done this (and realised I hadn't completely forgotten it -
> there was a TODO - I'd just underestimated *how* bad it would be).
> 
> also up at http://people.ubuntu.com/~robertc/baz2.0/knit-join-batching/.
> 
> Rob
> 

+1 from me

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060502/bd18a2d7/attachment.pgp