[RFC][DRAFT] Updated guide for upgrading to rich-roots [and the 2.0 beta formats]

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


...

> So about 1.5:1 the number of 'texts' we have to transfer now.	
> 
> time bzr branch qbzr/trunk test --no-tree
> 2.122s
> 
> time bzr branch qbzr-2a/trunk test --no-tree
> 5.000s
> 
> time bzr branch qbzr-2a-packed/trunk test --no-tree
> 2.637s
> 
> I have a possible patch which drops the 2.6s down to ~2.3s by changing
> how we parse the entry lines to extract the (file_id, revision_id) keys.
> However, I'm still poking at it a bit. Note that my "qbzr/trunk" branch
> has a single pack file (since that is what 'bzr branch' creates...)
> 
> John
> =:->
> 

Comparing this to branching bzr.dev

time bzr branch bzr-1.9
43.7

time bzr branch bzr-dev6
52.1

time bzr-chk-parser branch bzr-dev6
50.1

Shaving 2s out of 50 doesn't seem like much, though. I looked into
adding 'caching' into the parser. The idea is that as we walk these CHK
nodes most of it is redundant information that we have already parsed.
(If you compare 2 chk nodes, it is likely that only 1 line has actually
changed, but you have to parse all the other lines to find that one
difference.)

In testing the cache, I find it has a very good hit rate. Approximately
70-100:1 (for every 100 items parsed, only 1 of them isn't already in
the cache).

Now, it actually costs all the time that me might want to save, because
it has a dict lookup into the cache. And it doesn't save the PyMalloc
calls, because it creates a PyString and then runs it through the cache.
We also don't save much memory, etc, because all of these objects are
fairly short lived. Some make it into a set() but that will do its own
work to ensure only unique keys are present.

Some of it is simply the design of the chk map, and how we work with it.
With 1.9 formats, the inventory was *effectively* stored as a delta,
such that we would only read the line-delta and determine file_keys from
it. With chk, we don't have to read the entire inventory, but we *do*
have to read more than the minimal delta. By my caching numbers, that
seems to be about 70:1 more items that we read than the absolute minimum.


If I just add some 'time.clock()' calls in the appropriate spot, it
seems that "_filter_id_entry()" is taking ~17.5s out of the 50s (35%).
That is the time to deserialize the chk pages and parse them, and
extract the chk references and the text key references.

So there will be other overhead to do with more nodes being copied, and
more data being put into indexes, etc.


The only bit I can think of, would be to possibly push
'iter_interesting_nodes' down into Pyrex, and have it work on the raw
byte stream, rather than deserializing to LeafNodes and InternalNodes.
Though I'll note that lsprof claims only 9.4s is spent in
chk_map._deserialise. If that is close to accurate, then there is *some*
 time that can be trimmed, but we may want to look at other parts for
that 17s first.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAko2xgwACgkQJdeBCYSNAAOiBwCeLKn0l+0l7FDmEb6J7X0M9pYM
FOgAnjskPoQF5KFtPDT0C4IKf22hzS5y
=0ohs
-----END PGP SIGNATURE-----