CHKMap usage - optimizations?

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


...

>>> Most of the time is spent in CHKMap at the moment:
> 
>>> CHKMap.apply_delta(): 77.63%
>>>  * of which: Knit.add_lines(): 47.27%
>>> CHKMap.iteritems(): 5.14%
> 
>>> What's the best way of improving this performance ? Switching to packs?
> 
>> They will be faster.
> 
> What's the best place to be looking to get started with this? Can you
> give me a few hints?  bzrlib.plugins.search.index is a bit overwhelming..

So I'm wondering if this isn't the problem that transactions no longer
cache the knit object. So we might end up re-reading the entire .kndx
for every insert. I'm not positive to this, but if it is true, there
would be a huge boost just switching to packs. (They don't require
reading the whole index to access one entry.)

What are you using for the parent info for these entries? I'm just
thinking that CHK doesn't hint to knits/packs how to delta compress, as
we went with 'groupcompress' and hint a different way.

> 
>>> Is there some way I can use transactions here?
> 
>> apply_delta is pretty efficient. You are perhaps suffering some
>> thrashing on page re-reads, but the basic problem is likely to be kndx
>> related.
> The 30% that CHKMap.apply_delta() doesn't spend in Knit.add_lines()
> seems to be time spent purely in apply_delta itself. 

Sure, but if we make that take 0 time, it still is only 30% of your time
spent...

That said, there are a lot of places that could be the issue. Is there a
chance to get a profile? (I prefer --lsprof-file foo.txt since I don't
always have KCacheGrind available.)

> 
> Cheers,
> 
> Jelmer

So you might also look at how 'minimal' your deltas are. I know when I
was tuning the converter, having a small delta was quite beneficial.
(Often you can use different bases to generate a delta that results in
the same final content. I don't know if that is relevant for you.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoK3ggACgkQJdeBCYSNAAMemACghaxBUG/ZhGDaxQqB4v6eaFK0
vukAoLWtq3/BC6BLK7neefXguDkVAKkF
=5ml6
-----END PGP SIGNATURE-----