Bazaar: Out of memory

[Forest Bond]

Hi,

On Fri, May 09, 2008 at 09:56:13PM +0800, James Henstridge wrote:
> 2008/5/9 Ian Clatworthy <ian.clatworthy at internode.on.net>:
> > bazaar at ostkamp.fastmail.fm wrote:
> >
> >  >     parent_texts, left_matching_blocks, nostore_sha, random_id)
> >  >   File "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/knit.py",
> >  >   line 1035, in _add
> >  >     line_bytes = ''.join(lines)
> >
> >  Ah. That bug is now being encouraged in Bazaar itself, not in the
> >  fastimport plugin. It looks like the same bug though so the fix I
> >  applied may work there as well.
> >
> >  Try replacing line 1035 with these lines, taking care about the
> >  indentation (as Python requires):
> >
> >     lines_bytes = ''
> >     for line in lines:
> >         lines_bytes += line
> >
> >  FWIW, I'm pretty sure we're using the code we do because its faster.
> >  There's not much point to that though if it falls over on large files.
> >  I'd love to get this issue fixed ASAP in Bazaar's core so please let us
> >  know if it fixes your problem.
> 
> The .join() method is faster because it does less memory allocations
> and copies than the for loop you list above.  It should allocate a
> single string for the final concatenated string.  In contrast, the for
> loop version does a string allocation and copy for every iteration (so
> in the final iteration you'd expect it to have allocated roughly twice
> the memory).
> 
> If this change prevents MemoryErrors then something weird is going on.

Maybe this is helpful:

http://www.skymind.com/~ocrow/python_string/

-Forest
-- 
Forest Bond
http://www.alittletooquiet.net
http://www.pytagsfs.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20080509/7021fde7/attachment-0001.pgp