Bazaar: Out of memory

[James Henstridge]

2008/5/9 Ian Clatworthy <ian.clatworthy at internode.on.net>:
> bazaar at ostkamp.fastmail.fm wrote:
>
>  >     parent_texts, left_matching_blocks, nostore_sha, random_id)
>  >   File "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/knit.py",
>  >   line 1035, in _add
>  >     line_bytes = ''.join(lines)
>
>  Ah. That bug is now being encouraged in Bazaar itself, not in the
>  fastimport plugin. It looks like the same bug though so the fix I
>  applied may work there as well.
>
>  Try replacing line 1035 with these lines, taking care about the
>  indentation (as Python requires):
>
>     lines_bytes = ''
>     for line in lines:
>         lines_bytes += line
>
>  FWIW, I'm pretty sure we're using the code we do because its faster.
>  There's not much point to that though if it falls over on large files.
>  I'd love to get this issue fixed ASAP in Bazaar's core so please let us
>  know if it fixes your problem.

The .join() method is faster because it does less memory allocations
and copies than the for loop you list above.  It should allocate a
single string for the final concatenated string.  In contrast, the for
loop version does a string allocation and copy for every iteration (so
in the final iteration you'd expect it to have allocated roughly twice
the memory).

If this change prevents MemoryErrors then something weird is going on.

James.