[MERGE] performance enhancements for log -v

[Aaron Bentley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I've been working on improving performance for log -v, and I'm happy to
say I've succeeded.  It's gone from 2013 ms to 770 ms, i.e. it's 2.6
times faster.  Normal log operations are also faster;

To accomplish this, I've refactored the implementation of
KnitVersionedFile.  This actually brought back memories of reworking the
Arch builder, since both systems build up revisions using stored
full-texts, cached texts and deltas.

The main change was to retrieve the the inventories in batches, but
after I had that, I kept working on eliminating list construction, and
wound up knocking out a lot of the private methods.  The way a group of
texts is generated now is:

1. A dict called component_data is produced that holds describes data
about how to build all component versions.  This component_data doesn't
have the actual texts, but it knows where they're located in the knit

2. All the contents for all component versions are retrieved at once
using the component_data positions.  This allows us to use readv
efficiently.  A dict called record_map is produced based on
component_data, replacing position data with contents and digest.

3. Each requested revision is built.  Intermediate KnitContent objects
are held in a dict, and reused for building subsequent revisions, where
possible.

There's also a dict of *final* KnitContent objects, because texts
lacking final eols are handled very late in the process.  KnitComponents
without final eols cannot be reused to build later revisions from.

Additionally, since texts are generated for SHA1 verification, these are
kept in a dict and returned.  Most operations will use this map.

I don't think we can improve log performance much more without first
addressing knit record parsing.  According to lsprof, if get_texts takes
1.41 seconds, 0.85 of that is spent parsing records.

BEFORE CHANGES
...nchmarks.bench_log.LogBenchmark.test_cmd_log       OK    53ms/ 9447ms
...s.bench_log.LogBenchmark.test_cmd_log_subprocess   OK   598ms/10540ms
...nchmarks.bench_log.LogBenchmark.test_log           OK   410ms/10365ms
...hmarks.bench_log.LogBenchmark.test_log_screenful   OK    48ms/ 9148ms
...s.bench_log.LogBenchmark.test_log_screenful_line   OK    36ms/10362ms
....bench_log.LogBenchmark.test_log_screenful_short   OK    21ms/ 9096ms
...nchmarks.bench_log.LogBenchmark.test_log_verbose   OK  2013ms/11003ms
...nchmarks.bench_log.LogBenchmark.test_merge_log     OK   111ms/24973ms



AFTER CHANGES
...nchmarks.bench_log.LogBenchmark.test_cmd_log       OK    47ms/ 9215ms
......ench_log.LogBenchmark.test_cmd_log_subprocess   OK   322ms/ 9445ms
...nchmarks.bench_log.LogBenchmark.test_log           OK   403ms/ 9795ms
...hmarks.bench_log.LogBenchmark.test_log_screenful   OK    49ms/ 9185ms
...s.bench_log.LogBenchmark.test_log_screenful_line   OK    33ms/ 9458ms
....bench_log.LogBenchmark.test_log_screenful_short   OK    22ms/ 9232ms
...nchmarks.bench_log.LogBenchmark.test_log_verbose   OK   770ms/10521ms
...nchmarks.bench_log.LogBenchmark.test_merge_log     OK   110ms/24873ms

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEniFC0F+nu1YWqI0RAgmEAJ9afVaT/NhQ4P1R2Wr5nJznGIPjNQCeOBoZ
NtiKMAPMBRA+hbYZz3WhLCo=
=QFrk
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logv-fast.patch
Type: text/x-patch
Size: 21867 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060625/eaa78060/attachment.bin