Bazaar: Out of memory

[bazaar at ostkamp.fastmail.fm]

On Fri, 09 May 2008 18:08:42 +1000, "Ian Clatworthy"
<ian.clatworthy at internode.on.net> said:
> Ah. That bug is now being encouraged in Bazaar itself, not in the
> fastimport plugin. It looks like the same bug though so the fix I
> applied may work there as well.
> 
> Try replacing line 1035 with these lines, taking care about the
> indentation (as Python requires):
> 
>     lines_bytes = ''
>     for line in lines:
>         lines_bytes += line

Ian, I modified bzr sources as suggested and reinstalled it, 
also upgrade to Python 2.5.2, but it did not help.

There must be a memory leak in bazaar or the plugin, as I can 
see memory usage constantly growing up over time.

I would expect that after converting one changeset/commit from 
Git, the memory required for this step is freed up and reclaimed
by the Python garbage collector. Obviously this does not happen,
otherwise memory usage would just remain within a certain
range.

BTW: I spotted another location in the code similar to the stuff you
mentioned ~2209 or so, but I don't think this will help.
This code just builds up a string in pieces instead of all at once.
But IMHO the final outcome (storage occupied) will remain the same.

The current failure stack is now:

10:38:15 Found 30000 commits already loaded - skipping over these ...
10:45:55 Generated the file-ids cache - 9599 entries
bzr: ERROR: exceptions.UnboundLocalError: local variable 'line_bytes'
referenced before assignment

Traceback (most recent call last):
  File
  "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/commands.py",
  line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File
  "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/commands.py",
  line 802, in run_bzr
    ret = run(*run_argv)
  File
  "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/commands.py",
  line 504, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/ostkamp/.bazaar/plugins/fastimport/__init__.py", line 161,
  in run
    params, verbose)
  File "/home/ostkamp/.bazaar/plugins/fastimport/__init__.py", line 50,
  in _run
    return proc.process(p.iter_commands)
  File "/home/ostkamp/.bazaar/plugins/fastimport/processor.py", line 83,
  in process
    self._process(command_iter)
  File
  "/home/ostkamp/.bazaar/plugins/fastimport/processors/generic_processor.py",
  line 227, in _process
    processor.ImportProcessor._process(self, command_iter)
  File "/home/ostkamp/.bazaar/plugins/fastimport/processor.py", line
  105, in _process
    handler(self, cmd)
  File
  "/home/ostkamp/.bazaar/plugins/fastimport/processors/generic_processor.py",
  line 382, in commit_handler
    handler.process()
  File "/home/ostkamp/.bazaar/plugins/fastimport/processor.py", line
  171, in process
    self.post_process_files()
  File
  "/home/ostkamp/.bazaar/plugins/fastimport/processors/generic_processor.py",
  line 639, in post_process_files
    lambda revision_ids: self._get_inventories(revision_ids))
  File "/home/ostkamp/.bazaar/plugins/fastimport/revisionloader.py",
  line 60, in load
    text_provider)
  File "/home/ostkamp/.bazaar/plugins/fastimport/revisionloader.py",
  line 111, in _load_texts
    vfile.add_lines(revision_id, text_parents, lines)
  File
  "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/versionedfile.py",
  line 121, in add_lines
    left_matching_blocks, nostore_sha, random_id, check_content)
  File "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/knit.py",
  line 1007, in _add_lines
    parent_texts, left_matching_blocks, nostore_sha, random_id)
  File "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/knit.py",
  line 1038, in _add
    digest = sha_string(line_bytes)
UnboundLocalError: local variable 'line_bytes' referenced before
assignment

bzr 1.4 on python 2.5.2 (linux2)
arguments: ['/home/ostkamp/local/bin/bzr', 'fast-import', '-']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  fastimport           /home/ostkamp/.bazaar/plugins/fastimport
  [unknown]
  launchpad           
  /home/ostkamp/local/lib/python2.5/site-packages/bzrlib/plugins/launchpad
  [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Regards

Guido