SOLVED (was MemoryError in 2.2.0 in odd case: Can I reduce memory requirements?)

[Doug Lee]

Turns out I still run out of memory. :P

So... how do I compile extensions?  I don't find Pyrex for FreeBSD
4.11 anymore, I see no Cython port for it, and I think I'd need one of
those.  But it does sound like this needs to be my next step.

On Wed, Jan 12, 2011 at 10:09:19AM -0600, John Arbash Meinel wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 1/12/2011 9:51 AM, Doug Lee wrote:
> Finally got this server upgraded from Python 2.4 to Python 2.5; fixed
> the MemoryErrors.  Surprising, but I'll take it...

There are a few things that I know happened in 2.5. one of the biggest
was that Integer and Floating allocators could return memory back to the
pool. In 2.4 if you allocate 100k integers, you always had a pool for
100k integers even if you only were using 1 of them from there on out.

So doing:

i = range(10**6)
j = map(float, i)
del i
del j

Would still have 1 million PyInt objects and 1 million PyFloat objects
allocated from the system. Creating a new int would re-use one of these
objects, but the mem was never returned to be used elsewhere.

off-hand, I don't know anything that specifically caused this, but it is
possible.

The other major factor is whether you have compiled extensions.
cElementTree doesn't matter much anymore, but bzrlib._groupcompress_pyx
can make a huge difference on memory consumption.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0t0i8ACgkQJdeBCYSNAAPLWACeJpLEm+FQysvGkGXyzh2iRtVa
fcgAoLeFsbWNNV3I1FrZKAvDKg1EHc+O
=bUMS
-----END PGP SIGNATURE-----

-- 
Doug Lee                 dgl at dlee.org                http://www.dlee.org
SSB BART Group           doug.lee at ssbbartgroup.com   http://www.ssbbartgroup.com
"I honestly believe it is better to know nothing than to know what ain't so."
- Josh Billings, 1818-1885 (in "Solemn Thoughts")