[MERGE][1.6.1] 1.6 fetch regression

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Arbash Meinel wrote:
> Sending again because I forgot to say "merge".
> 
> John Arbash Meinel wrote:
>> John Arbash Meinel wrote:
>> ...
> 
>>> My best guess is that we are getting some really long buffered bytes, and the
>>> cost of reallocating that is costing us a lot.
> 
> This ended up being a (the?) major factor.
> Specifically, in the StatefulDecoder we would buffer up the bytes by doing:
> 
> self._in_bytes += bytes
> 
> Which would cause us to reallocate large (MB+) strings.
> 
> And then in the readv parser, we would do:
> 
>  data = data[consumed:]
> 
> Which would cause us to downsize the strings. Notice, though, that it requires
> 2x memory while doing it, because it has to create a new string, copy the
> bytes over, and then destroy the original.
> 
> My initial work on fixing the "input" side failed, because we peek many times
> at the string while waiting for more bytes to come. But I've since worked
> around that.
> 
> The attached patch drops the time for "bzr branch bzr+ssh://localhost/bzr.dev"
> from 1m26s to 39s.
> 
> $ old
> VmPeak:	  202736 kB
> bzr -Dmemory branch  xxx  54.60s user 30.11s system 96% cpu 1:27.58 total
> $ new
> VmPeak:	  181136 kB
>  33.64s user 0.84s system 91% cpu 37.640 total
> 
> Notice the *huge* difference in "system" time.
> 

Attached is a slight update. I had left some "assert" statements in the code,
and one other small test was failing.

I would also mention that for "bzr branch lp:mysql-server" it changed the time
from 80+ minutes down to ~23minutes.

We are only really seeing this now because Robert's work made it easier for us
to talk about more data at a time, and this code path is probably at least
O(N^2) by length. (If you read data at fixed size pages you have to read L/C
pages, and you reallocate the buffer each time.)

I've decided to go ahead and do a few bug fixes and get them into a 1.6.1
release. There are 2 "critical" bugs that will also make it into 1.6.1.

https://bugs.edge.launchpad.net/bzr/+bug/261339
and
https://bugs.edge.launchpad.net/bzr/+bug/262333

I'm hoping to get some patches reviewed and 1.6.1rc1 out tomorrow.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIt1gbJdeBCYSNAAMRAoyCAJ0Ri0zSDStnuuUE3+O+EslMO+aj6ACfcNJT
b+4YILlyfpxEyi/1UJQq9JQ=
=FK95
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hpss_readv.patch
Type: text/x-diff
Size: 24571 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20080828/471947a0/attachment-0001.bin