knits - slow sftp pushes

[Martin Pool]

On 02/05/2006, at 1:06 AM, John Arbash Meinel wrote:

> I think the join() should indeed work by doing buffering. I don't know
> exactly how the code is working, but shouldn't join be doing  
> everything
> in memory, and then flushing to disk when it is done?
>
> I believe that is how the old weave code worked where you would  
> build up
> your Weave, and then tell it to write to disk. But I didn't follow the
> VersionedFile stuff enough to know if they kept that format.

Knits have more integration between the in-memory and the on-disk  
form: part of the point is that you can read and write them without  
pulling the whole thing into memory.  So when you add a record, at  
the moment they incrementally write it out straight away.  There are  
various ways they can be smarter about batching things up -- probably  
integrating it with transaction buffering would be good eventually.

-- 
Martin