MemoryError on commit with large file

Joel Hardi joel at hardi.org
Tue Oct 2 04:10:54 BST 2007


Hi all--

I'm a newcomer (SVN refugee), and just installed 0.91 for the first  
time. It failed with a MemoryError exception on initial commit while  
trying to commit a 301 MB file on a system with 1 GB RAM.

 From previous similar posts to the list, I gather that commits  
require 2-3 copies of the file to fit in memory at once, which is  
obviously very fast for tiny bits of source code but not appropriate  
for lots of other applications (I was copying a database, but  
graphics and video projects also often run into the 100s of MB-100s  
of GB range).

Are there plans to accommodate arbitrarily large files, either by  
diffing using file streams or just not diffing files at all if they  
meet some size threshold, match a glob like *.dv or when they throw a  
MemoryError (I'd be perfectly happy with that since these are binary  
files that I don't need to diff)?

FYI, I've also been running Mercurial on the same file set for the  
past week on the same system; it allows me to commit and make local  
branches on these files, but runs out of memory on network copies,  
which is what brought me to bzr.


bzr: ERROR: exceptions.MemoryError:

Traceback (most recent call last):
   File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line  
800, in run_bzr_catch_errors
     return run_bzr(argv)
   File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line  
758, in run_bzr
     ret = run(*run_argv)
   File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line  
492, in run_argv_aliases
     return self.run(**all_cmd_args)
   File "/usr/lib/python2.4/site-packages/bzrlib/builtins.py", line  
2323, in run
     author=author)
   File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line  
165, in write_locked
     return unbound(self, *args, **kwargs)
   File "/usr/lib/python2.4/site-packages/bzrlib/workingtree_4.py",  
line 248, in commit
     result = WorkingTree3.commit(self, message, revprops, *args,  
**kwargs)
   File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line  
165, in write_locked
     return unbound(self, *args, **kwargs)
   File "/usr/lib/python2.4/site-packages/bzrlib/mutabletree.py",  
line 211, in commit
     revprops=revprops, *args, **kwargs)
   File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line  
321, in commit
     self._update_builder_with_changes()
   File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line  
653, in _update_builder_with_changes
     self._populate_from_inventory(specific_files)
   File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line  
725, in _populate_from_inventory
     parent_id, definitely_changed, existing_ie)
   File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line  
776, in _record_entry
     path, self.work_tree)
   File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line  
2245, in record_entry_contents
     ie.snapshot(self._new_revision_id, path, previous_entries, tree,  
self)
   File "/usr/lib/python2.4/site-packages/bzrlib/inventory.py", line  
745, in snapshot
     self.text_sha1, self.text_size)
   File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line  
2291, in modified_file_text
     file_parents.keys())
   File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line  
2308, in _add_text_to_weave
     result = versionedfile.add_lines_with_ghosts(
   File "/usr/lib/python2.4/site-packages/bzrlib/versionedfile.py",  
line 124, in add_lines_with_ghosts
     parent_texts, nostore_sha)
   File "/usr/lib/python2.4/site-packages/bzrlib/knit.py", line 838,  
in _add_lines_with_ghosts
     parent_texts, None, nostore_sha)
   File "/usr/lib/python2.4/site-packages/bzrlib/knit.py", line 931,  
in _add
     access_memo = self._data.add_record(version_id, digest,  
store_lines)
   File "/usr/lib/python2.4/site-packages/bzrlib/knit.py", line 2020,  
in add_record
     size, sio = self._record_to_data(version_id, digest, lines)
   File "/usr/lib/python2.4/site-packages/bzrlib/knit.py", line 1997,  
in _record_to_data
     ["end %s\n" % version_id]))
   File "/usr/lib/python2.4/site-packages/bzrlib/tuned_gzip.py", line  
330, in writelines
     self.write(''.join(lines))
   File "/usr/lib/python2.4/site-packages/bzrlib/tuned_gzip.py", line  
322, in write
     self.fileobj.write( self.compress.compress(data) )
MemoryError

bzr 0.91.0 on python 2.4.4.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'commit', '-m', 'Added initial projects  
and files.']

** please send this report to bazaar at lists.ubuntu.com




More information about the bazaar mailing list