[MERGE][bug #300289] Avoid extracting revision texts

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The attached patch is yet another one that improves the fetch from
stacked branches into a standalone branch.

It turned out that a significant portion of time was spent in extracting
the revision gzipped text back into a fulltext string and then packing
it back down into a gzipped text again.

I did manage to reproduce the explicit circumstances by forcing the
record stream order. I was able to create a knit that had the right
properties, but it turns out that fetches from knits => pack are in
iter(dict) order. (I would have expected it to at least sort by position
offset, but that does not seem to be the case.)

Anyway, this patch changes the code so that instead of passing
"include_delta_closure=True" instead it does a topological sort of the
revisions and forces them to be in topo order.

Testing it locally, the topo sort isn't measurable in the actual
performance (1m15s with this patch, 1m15s with unordered and
include_delta=False).

So with this patch and the earlier "iter_entries"/"difference_update"
patch, the time to branch from a stacked is down to 1m15s, rather than
45s for a regular pack=>pack fetch, and 3m9s without my patches for a
fetch from stacked.

The remaining time seems to be primarily caused by double-handling the
inventory. Also in the future, we might see some time spent because the
'Index.add_records()' checks to make sure the revision id doesn't
already exist in the repository. So if you had 10,000 revisions, and
were fetching another 10,000 you would end up searching all of your
local graph to see that it really is true that nothing is there.
However, for initial branch, it seems that looking up 25,000 keys in an
empty repository isn't too painful.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkksxtAACgkQJdeBCYSNAAME6wCfTgMV8l1xY4WtXLMoY7UmAU0m
brYAoMhu6m0jaQRj7tS+TD1sNdyPl7A+
=oV+U
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: revision_stream.patch
Url: https://lists.ubuntu.com/archives/bazaar/attachments/20081125/5d34bdc3/attachment.diff