BTree + CHK Inefficiencies

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Szakmeister wrote:
> On Thu, Aug 5, 2010 at 12:30 PM, John Arbash Meinel
> <john at arbash-meinel.com> wrote:
> [snip]
>> Yeah, it is actually written for checkout, but I could trigger the base
>> bug using 'bzr ls -R' without having to actually extract file content.
> 
> Very cool.  I might grab the code and try this one on of our larger
> repositories.  Right now, we had to take a different approach to the
> problem because checkout would fail on most of our boxes--it consumed
> several GB of memory, which ultimately caused the C extension to fail.
>  Disabling the C extension would allow it to succeed eventually, but
> you needed 3+ GB of memory available--which was difficult to come by
> since we're all running VMs--not to mention it was pretty slow.
> 
> BTW, thanks for posting all of the tidbits about how memory is
> consumed and such.  It helps poor saps like me get to know the
> internals better without having looked at actual code.
> 
> -John
> 

Well, if you are interested in helping out, you can do the checkout with
extensions, and get a memory dump.
You'll need 'Meliae' which is my memory debugging python library. 'bzr
branch lp:meliae'.

Before it crashes interrupt the process (Ctrl+\ on Linux "Break"
(Ctrl/Fcn/+ Pause) on Windows). And then you can run

  from meliae import scanner
  scanner.dump_all_objects('checkout.dump')

That will take a little bit of memory itself (at 8M objects, it takes
about 120MB, if you're even bigger, you'll want to allow for that).

You'll then want to compress it (gz/bz2, etc) and I can take a look at
it. It may expose some of the content of your tree, so if that is an
issue, we can think about what to do. (Strings are truncated at 100
bytes, though, so it can't expose very much.)

Now, it is possible that the memory consumed is actually because of
individual file content, and not what I'm doing here (which is more
about lots of inventory data, aka lots of small files).

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxbbkUACgkQJdeBCYSNAAPQuwCfdhkoIZ4yhZg3qxui8dKcXH+T
LAUAoI6pncRUOljmqTv0DZJB211/Ddsb
=HfiA
-----END PGP SIGNATURE-----