[RFC] Faster load of full inventory for development6-rich-root?

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ian Clatworthy wrote:
> Partial inventory processing is now much faster in
> development6-rich-root than previous formats. In general, we want to
> avoid processing a full inventory, but there are times when we need to
> and, unfortunately, that's terribly slow right now. :-(
> 
> Benchmarking on OOo with 75k files shows two cases in particular:
> 
> * 97% of bzr send is taken up in walking 2 inventories: one to find
>   what to include in the bundle and one to calculate the testament_sha1
>   value of the revision to merge.

"1 to find what to include in the bundle" is this a complete walk? Or
just an iter_changes walk? Certainly it doesn't seem like we should have
to walk the whole thing.

"one to calculate the testament_sha1" this is a bigger issue, as we
probably *have* to walk everything.

I'm curious how my "move recent to front" patch would effect these sorts
of times, especially with a prototyped "batch requests".

> 
> * 99% of "bzr ls -1 --recursive" is iter_entries_by_dir().
> 
> From memory, John had an 'all-children' patch that he was playing with?
> Is that the right way forward? If not, any other ideas?
> 
> Ian C.
> 
> 

Just from gut feeling, the time to read chk pages is quite a bit too
high. I don't know exactly what it should be, but just considering how
much more expensive it should be than reading an XML inventory of the
same, it shouldn't be as different as it is now.

I'm curious with something like OOo, if the BTreeIndex overhead isn't
killing us. This could be ameliorated somewhat if we knew we were
walking everything (like iter_entries_by_dir()) by telling the subsystem
to read everything. Because then it can read the root page, 255 children
pages, 65k next pages, etc. I would at least guess that OOo is has
enough chk pages that getting cache coherency is really difficult.

I'm also curious if OOo is big enough to cause us to do a 3-level deep
inventory (1 root level, 1 internal, 1 leaf). Care to run 'bzr
repository-details' on your OOo conversion and paste the output?

John
=:->


John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkom6A4ACgkQJdeBCYSNAAOk+gCeLJ3rtCBIDtbwJ31LyWKwr+yR
SGIAoKcE7wqO9LsyzqhaWhNL0mrEZcpG
=uE0Y
-----END PGP SIGNATURE-----