[MERGE] Cache inventory.knit so we don't download it twice

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Ellerman wrote:
> On 7/14/06, John Arbash Meinel <john at arbash-meinel.com> wrote:

...

> 
> 
> I've done a bit of profiling with this patch (including spill to
> disk), vs bzr.dev and vs. my KnitData.read_records_iter patch. Graph
> attached.
> 
> In short it doesn't seem to have much effect on memory usage. What the
> graph does show, IMHO, is that my read_records_iter() in arbitrary
> order patch is Good Stuff (TM), and we should work out what changed
> recently to cause the memory spike at the end of branching.
> 

As a public API, I think it should return in the order requested, if
only because that is what we have specified so far.
However, the only callers seem to be internal functions at this point.

read_records_iter_raw() seems to only do a little bit of caching, rather
than building up a map, and then iterating.

read_records_iter() is building up a map, and then iterating, and as I'm
sure Michael noticed, the only callers at this point actually build up
their own map, and then return it.

So some of our inefficiency is probably because we are building up 2
maps. And I'm guessing the version ids are not interned, so we end up
holding a lot of copies of the version ids in memory.

For api compatibility, though, I think we can create a
'read_records_iter_unsorted', which does what Michael suggests, and then
read_records_iter, can just thunk into that, and return what it can.

The attached diff is against my 'cache-inventory' branch, with spill to
disk enabled. But if Michael can run it against his large branches and
it drops the memory consumption, I'm willing to clean it up for use with
bzr.dev.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEvRUuJdeBCYSNAAMRAgfzAKChWFgPr3hDg21k037i20Tq80Qs+QCgzE7f
046jbnGb9P3B6bYAIY2xr2g=
=N10q
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iter_unsorted.diff
Type: text/x-patch
Size: 5197 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060718/e91ddb0b/attachment.bin