[bbc:MERGE] "look before you leap" during serialization

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So the way chk_map nodes are written to the repository, is that we build
up the lines, and just insert them. And then if we find a collision
later, we semi-ignore it. (generally we don't write the collision to the
index, so a future autopack, etc, will filter it out.)

However, when doing conversions, a *lot* of the pages are collisions.
Especially with the random-base-id generic fetch code.

This patch changes it so that we compute the sha1 sum (after all, we
already have the bytes we will be inserting), and does the has_key check
before calling add_lines.

I'll also note that add_lines() was already doing the check, so this
isn't adding a check, it is just moving it into the serialise()
function. (I also pass random_id=True so that the check doesn't happen
again.)

The difference for converting the first 1043 revisions of a mysql
repository is 42MB versus 28MB. (In other words, there are a *lot* of
colliding pages in there.)

This isn't as necessary with my updated generic fetch code (being sent
after this email). With better selections of base trees, the conversion
drops to 30MB.

This doesn't improve conversion speed much (maybe 2m30s => 2m20s), but
it certainly makes me feel better to not be writing 2x the amount of
data, just to throw it away.

Part of this also came from my "knit-delta compression" work, where it
was more important to not write out the delta with a potentially
different set of parents. However, I still like the patch, though I
wanted to make sure Robert was okay with it.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmcT9IACgkQJdeBCYSNAANvYACfeTIaAebVMnoyknJjLNgs6Njl
RlMAnjuhFMumZUIoO3gN/oAlwysV/gSY
=pZkU
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: look_before_you_leap.patch
Url: https://lists.ubuntu.com/archives/bazaar/attachments/20090218/d1433f68/attachment-0001.diff