2GB limit

[Maritza Mendez]

On Tue, Oct 12, 2010 at 9:01 AM, Maritza Mendez <martitzam at gmail.com> wrote:

> Well that is strange.  The repo in question was freshly imported.
> Revno 1.  I note that the client machine which reported the error has
> 2GB of ram but the server machine has only 512MB and the peak RSS for
> the bzr serve process was 500 MB just before the error.
>
> I see that some MemoryError diagnostics landed a few hours ago.
> Should be interesting when I get some time to try.
>
> Thanks for the I insight into how to think about bzr memory use.
>
> ~M
>
> On 10/12/10, John Arbash Meinel <john at arbash-meinel.com> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> >
> >> Ok.  Thanks for confirming that I have a test case here, based on the
> >> groupcompress messages.
> >
> >> Now here's something.  The branch in question has 1.25 GB distributed
> >> (very unevenly) over 700 files.  The largest single file is 195MB and
> >> the cumulative size distribution looks like this:
> >
> >>    Size(MB)    Total Number of Files Larger than Size
> >>    =======    ============================
> >>    100               2
> >>     10             15
> >>      1            173
> >
> >> The second largest file is ~120MB.  So could the ca, 419MB number come
> >> from a pack which is combining texts?
> >
> >> Thanks
> >> ~M
> >
> > I can't say for sure. It is unlikely for us to combine 2 file histories
> > into one group if it is larger than 4MB. The current logic should be:
> >
> > a) If the current group has all the same file, it is allowed to grow to
> >    2x the size of the largest content or 4MB, whichever is larger.
> > b) If a group has mixed content, it is capped at 2MB
> >
> > So if you have 10,000 revisions of a 1KB file (with no delta
> > compression), it should pack into 3 groups (4MB, 4MB, 2MB).
> >
> > If you have 10 files that are all 1MB (no delta compression) it will be
> > in 5 groups (2MB, ...)
> >
> > If you have 3 versions of an ISO and get some delta compression, it
> > should end up in a very large group (700MB base, + <700MB of deltas).
> >
> > Now it is possible that the "419" includes the compressed and
> > uncompressed sizes. Though that is still only 240MB...
> >
> > John
> > =:->
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.9 (Cygwin)
> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> >
> > iEYEARECAAYFAky0bOYACgkQJdeBCYSNAAMlaACdG3MNEtruBYhaN/QWcsmthvgU
> > cp4AmwQt9L7fX29A05m6TfR45qWKrhwP
> > =Hpin
> > -----END PGP SIGNATURE-----
> >
>

John: I thought about what you wrote and came up with an experiment to help.

This thread is getting pretty long, but I have new data on the repo which is
tripping the OOM error.
Again, the trunk is hosted on a lowly 512MB Lucid box.

I then did a clean install of the stand-alone windows installer (2.2.1) on
both a WinXP32 and a Win7x64 box.

The OOM error occurs on a WinXP32 box with 2GB RAM when branching from the
trunk:
   - peak memory is 501MB, pagefaults=2.01million, just before the OOM
error.

But!  The error does not occur on a Win7x64 box with 4GB of RAM when
branching from the trunk:
   - peak memory is 870MB, pagefaults=2.38million, and the branch succeed

And although it is risky to compare WinXP32 to Win7x64, I now believe that
the real problem is not that I'm hitting the actual 2GB limit but that there
isn't enough contiguous memory left to give python the 870MB it needs.
Notice that the peak memory is about 350K until the "adding file contents"
stage starts.  Then the peak memory skyrockets, along with the pagefaults of
course.

So... step 1 is have enough memory to go around.  Step 2 is to be able to
use 64-bit code when the problem really is the 2GB limit.   Happily, I got
bzr extensions built for x64 yesterday (but it needs testing).

~M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/bazaar/attachments/20101017/e09e35c1/attachment-0001.htm