[Bug 336890] Re: MemoryError in pack.bytes_record during fetch of enormous repository

[Jelmer Vernooij]

** Tags removed: check-for-breezy

** Also affects: brz
   Importance: Undecided
       Status: New

** Changed in: brz
       Status: New => Triaged

** Changed in: brz
   Importance: Undecided => Low

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to bzr in Ubuntu.
https://bugs.launchpad.net/bugs/336890

Title:
  MemoryError in pack.bytes_record during fetch of enormous repository

Status in Breezy:
  Triaged
Status in Bazaar:
  Confirmed
Status in bzr package in Ubuntu:
  Triaged

Bug description:
  Binary package hint: bzr

  When using sftp to check out my ginormous repository of PDFs (physics
  articles I need to keep synchronized between desktops, netbook, and
  n810), bzr eats all of my available RAM (gets up to 1.4-1.5GB) and
  then, of course, runs out of RAM and gets killed:

  $ bzr branch sftp://<workstation>./home/<user>/Documents
  /usr/lib/python2.6/dist-packages/Crypto/Hash/SHA.py:6: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
    from sha import *
  /usr/lib/python2.6/dist-packages/Crypto/Hash/MD5.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
    from md5 import *
  bzr: ERROR: exceptions.MemoryError:                                            

  Traceback (most recent call last):
    File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 896, in run_bzr_catch_errors
      return run_bzr(argv)
    File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in run_bzr
      ret = run(*run_argv)
    File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 542, in run_argv_aliases
      return self.run(**all_cmd_args)
    File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 1045, in run
      source_branch=br_from)
    File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 1116, in sprout
      result_repo.fetch(source_repository, revision_id=revision_id)
    File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1118, in fetch
      find_ghosts=find_ghosts)
    File "/usr/lib/python2.6/dist-packages/bzrlib/decorators.py", line 192, in write_locked
      result = unbound(self, *args, **kwargs)
    File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 2985, in fetch
      return self._pack(self.source, self.target, revision_ids)
    File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 2992, in _pack
      revision_ids).pack()
    File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 655, in pack
      return self._create_pack_from_packs()
    File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 809, in _create_pack_from_packs
      self._copy_text_texts()
    File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 786, in _copy_text_texts
      self.new_pack.text_index, readv_group_iter, total_items))
    File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 900, in _copy_nodes_graph
      write_index, output_lines, pb, readv_group_iter, total_items):
    File "/usr/lib/python2.6/dist-packages/bzrlib/repofmt/pack_repo.py", line 944, in _do_copy_nodes_graph
      pos, size = writer.add_bytes_record(raw_data, names)
    File "/usr/lib/python2.6/dist-packages/bzrlib/pack.py", line 154, in add_bytes_record
      serialised_record = self._serialiser.bytes_record(bytes, names)
    File "/usr/lib/python2.6/dist-packages/bzrlib/pack.py", line 105, in bytes_record
      return ''.join(byte_sections)
  MemoryError

  bzr 1.12 on python 2.6.1 (linux2)
  arguments: ['/usr/bin/bzr', 'branch', 'sftp://<workstation>/home/<user>/Documents']
  encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
  plugins:
    avahi                /usr/lib/python2.6/dist-packages/bzrlib/plugins/avahi [0.3dev]
    builddeb             /usr/lib/python2.6/dist-packages/bzrlib/plugins/builddeb [2.1dev]
    bzrtools             /usr/lib/python2.6/dist-packages/bzrlib/plugins/bzrtools [1.12]
    dbus                 /usr/lib/python2.6/dist-packages/bzrlib/plugins/dbus [unknown]
    gtk                  /usr/lib/python2.6/dist-packages/bzrlib/plugins/gtk [0.95.0.final.1]
    launchpad            /usr/lib/python2.6/dist-packages/bzrlib/plugins/launchpad [unknown]
    loom                 /usr/lib/python2.6/dist-packages/bzrlib/plugins/loom [1.4dev]
    netrc_credential_store /usr/lib/python2.6/dist-packages/bzrlib/plugins/netrc_credential_store [unknown]
    rebase               /usr/lib/python2.6/dist-packages/bzrlib/plugins/rebase [0.4.2]
    search               /usr/lib/python2.6/dist-packages/bzrlib/plugins/search [1.7dev]
    stats                /usr/lib/python2.6/dist-packages/bzrlib/plugins/stats [unknown]
    svn                  /usr/lib/python2.6/dist-packages/bzrlib/plugins/svn [0.5]
  *** Bazaar has encountered an internal error.
      Please report a bug at https://bugs.launchpad.net/bzr/+filebug
      including this traceback, and a description of what you
      were doing when the error occurred.

  This occurred while it was "Adding file contents"

  This occurs both when I do a lightweight checkout (bzr checkout
  --lightweight) and when doing a branch (bzr branch).  I'd much prefer
  the lightweight checkout; then I only need rev history in one location
  and don't have to duplicate the files all over.

To manage notifications about this bug go to:
https://bugs.launchpad.net/brz/+bug/336890/+subscriptions