[MERGE] Fetch tweaks

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
| This allows repositories more control over their fetch operations in the
| generic fetching code. Doing so allows the groupcompress format to avoid
| having to figure out full text representations, rather getting
| everything as fulltext in the first place; and eliminates an unnecessary
| reconcile post-fetch.
|
| -Rob
|

BB:approve

I like this patch as it stands, though with one caveat. Specifically
during first branch, passing _fetch_uses_deltas = False will read the
entire repository into memory. It will be somewhat efficient, in that it
will share strings in the in-memory lists, up until the point that you
actually fetch a bit of text.

Then it does ''.join(lines) which doubles memory consumption for that
text (while still caching the original lines). If the caller doesn't
hang onto the text it will probably be ok.

To truly scale up, we need to change the 'get_record_stream()' code that
blindly unpacks all of the requested keys so that we only unpack a few
at a time. I don't have a good answer for that, as how do you decide how
much to unpack for efficiency versus memory consumption.

Anyway, this is still better than what we have (as it lets us experiment
with it), and it shouldn't change the behavior of anything *today*.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkiN6mQACgkQJdeBCYSNAAOuXACgpxXbaWtO2wbTO9/ZFUB0IFeD
d+UAn2PDZmQbzka2GdmpUllJ6hLrxsB8
=ck8R
-----END PGP SIGNATURE-----