[MERGE] Re: Accelerating push via bzr:// and friends

[Martin Pool]

On Wed, May 21, 2008 at 1:52 PM, Andrew Bennetts <andrew at canonical.com> wrote:

> We're already insulated from dangerously large send()s because we use
> osutils.send_all, which breaks up large writes.

OK.  I hadn't realized we were still going through that layer.  So,
you're right that we do avoid that particular error.  There are still
some problems though:

We now have two layers of buffering, which can't be good for
performance or memory use, or code clarity for that matter.  Maybe
this code should call sendall directly.

Also, we can't test that we're avoiding over-large sends, because
send_all goes direct to the socket.  Your new tests would be an ideal
place to do it directly.

But obviously this would be a big improvement as is, so perhaps you
could just add a todo about this.

> If something sends a large monolithic body, then the memory consumption will be
high.  On the other hand, that's going to be true already.  The solution is to
not hold very large strings in memory to start with, I think.

You are right, I meant to acknowledge that in my mail but forgot.
Still, we are now using twice the amount.

-- 
Martin <http://launchpad.net/~mbp/>