[MERGE] Cache inventory.knit so we don't download it twice

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Ellerman wrote:

...

> 
> Ok, you missed the most important call to read_records_iter(),
> although perhaps that's because you thought it need to be in order?
> The one in iter_lines_added_or blah blah.
> 
> I hacked your patch to add it:
> 
> === modified file 'bzrlib/knit.py'
> --- bzrlib/knit.py      2006-07-17 23:48:11 +0000
> +++ bzrlib/knit.py      2006-07-19 05:16:28 +0000
> @@ -807,7 +807,7 @@
>         try:
>             pb.update('Walking content.', count, total)
>             for version_id, data, sha_value in \
> -                self._data.read_records_iter(version_id_records):
> +                self._data.read_records_iter_unsorted(version_id_records):
>                 pb.update('Walking content.', count, total)
>                 method = self._index.get_method(version_id)
>                 version_idx = self._index.lookup(version_id)
> 

Unfortunately because of how the content is walked, I believe we really
*do* need to iterate in order.
That iter_lines_added... has an api decision that we always walk the
history in an explicit order. Otherwise when we annotate we can get
different results. (If a line is repeated, it may match differently if
history is sorted in a different order).

Attached is another possible patch, though. Basically, it still returns
in requested order. Only now it only queues up items that were requested
out of order. I use a similar trick for my readv() optimizations. And
with readv() I found it worked quite well to keep the queue small.

I'm hoping to find something similar here.

> 
> And with that it looks good. There's still a nasty spike towards the
> end of the branching that I'd like to tie down.
> 
> Graph attached.
> 
> cheers

Are any of these giant branches public, so that someone like myself
could try and track them down locally?
Also, what are you using to profile the memory consumption? I think it
would be a useful tool to have.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEvnHDJdeBCYSNAAMRAoLLAKCPAtD9ppFcjFy2Wny6EvO8/JGxAgCgm71s
p0rpOzORHnqA1SKAMY7AfUk=
=KBfJ
-----END PGP SIGNATURE-----