[MERGE] Cache inventory.knit so we don't download it twice

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Ellerman wrote:
> On Wed, 2006-07-19 at 20:32 +0200, Jelmer Vernooij wrote:
>> On Wed, 2006-07-19 at 12:54 -0500, John Arbash Meinel wrote:
>>> Are any of these giant branches public, so that someone like myself
>>> could try and track them down locally?
>>> Also, what are you using to profile the memory consumption? I think it
>>> would be a useful tool to have.
>> There are bzr copies of the two main Samba branches converted using
>> bzr-svn at 
>> http://people.samba.org/bzr/jelmer/samba/3.0/ and
>> http://people.samba.org/bzr/jelmer/samba/4.0/.
>>
>> The 4.0 one has a little over 9000 revisions, and there's a bit under
>> 2000 files in those trees IIRC.
> 
> I've been using an earlier copy of the Samba branch, as well as a kernel
> tree with no history.

Well, I just grabbed the same Samba 4.0 branch, and did these profiles,
using bzr.dev, my latest version of the inventory caching, and one that
uses read_iter_..._unsorted.

I can see that my work of only caching what we have to does help, as the
memory is slightly less, and drops down earlier. But it seems that we
are actually requesting versions in a very bad order, considering the
big drop isn't until very late in the game.

> 
> To profile the memory usage I just do a dodgy shell hack:
> 
> while true; do ps -eo rss,args | awk '/[b]zr branch/ {print $1}'; sleep
> 1; done
> 
> And then run it through quickplot(.sf.net) or OOO.
> 
> cheers
> 

I created a python wrapper which spawns a program, and then uses the
pid and uses /proc/<pid>/status. I'm sampling every 0.1 seconds, but my
plot looks pretty much identical to yours.

So I'm able to reproduce your results, including the peak at the end.
And since Robert has stated that the order doesn't matter. And I also
saw that the docstring says the same thing, it seems reasonable to
switch to using read_iter_unsorted everywhere, and possibly deprecating
read_iter.

I want a little bit of time to play with a few things, but it seems
reasonable to switch to unsorted all the way around.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEv4OtJdeBCYSNAAMRAiYLAKDI9N53cYAEM5nVMSfEs/Tg/p8wbgCgi6Oq
Q3jVZhZz88cgIX/tPhYOems=
=t2oz
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem-consumption.png
Type: image/png
Size: 6169 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060720/5a3b2dbf/attachment.png