MemoryError on commit with large file
Joel Hardi
joel at hardi.org
Tue Oct 2 04:10:54 BST 2007
Hi all--
I'm a newcomer (SVN refugee), and just installed 0.91 for the first
time. It failed with a MemoryError exception on initial commit while
trying to commit a 301 MB file on a system with 1 GB RAM.
From previous similar posts to the list, I gather that commits
require 2-3 copies of the file to fit in memory at once, which is
obviously very fast for tiny bits of source code but not appropriate
for lots of other applications (I was copying a database, but
graphics and video projects also often run into the 100s of MB-100s
of GB range).
Are there plans to accommodate arbitrarily large files, either by
diffing using file streams or just not diffing files at all if they
meet some size threshold, match a glob like *.dv or when they throw a
MemoryError (I'd be perfectly happy with that since these are binary
files that I don't need to diff)?
FYI, I've also been running Mercurial on the same file set for the
past week on the same system; it allows me to commit and make local
branches on these files, but runs out of memory on network copies,
which is what brought me to bzr.
bzr: ERROR: exceptions.MemoryError:
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line
800, in run_bzr_catch_errors
return run_bzr(argv)
File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line
758, in run_bzr
ret = run(*run_argv)
File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line
492, in run_argv_aliases
return self.run(**all_cmd_args)
File "/usr/lib/python2.4/site-packages/bzrlib/builtins.py", line
2323, in run
author=author)
File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line
165, in write_locked
return unbound(self, *args, **kwargs)
File "/usr/lib/python2.4/site-packages/bzrlib/workingtree_4.py",
line 248, in commit
result = WorkingTree3.commit(self, message, revprops, *args,
**kwargs)
File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line
165, in write_locked
return unbound(self, *args, **kwargs)
File "/usr/lib/python2.4/site-packages/bzrlib/mutabletree.py",
line 211, in commit
revprops=revprops, *args, **kwargs)
File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line
321, in commit
self._update_builder_with_changes()
File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line
653, in _update_builder_with_changes
self._populate_from_inventory(specific_files)
File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line
725, in _populate_from_inventory
parent_id, definitely_changed, existing_ie)
File "/usr/lib/python2.4/site-packages/bzrlib/commit.py", line
776, in _record_entry
path, self.work_tree)
File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line
2245, in record_entry_contents
ie.snapshot(self._new_revision_id, path, previous_entries, tree,
self)
File "/usr/lib/python2.4/site-packages/bzrlib/inventory.py", line
745, in snapshot
self.text_sha1, self.text_size)
File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line
2291, in modified_file_text
file_parents.keys())
File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line
2308, in _add_text_to_weave
result = versionedfile.add_lines_with_ghosts(
File "/usr/lib/python2.4/site-packages/bzrlib/versionedfile.py",
line 124, in add_lines_with_ghosts
parent_texts, nostore_sha)
File "/usr/lib/python2.4/site-packages/bzrlib/knit.py", line 838,
in _add_lines_with_ghosts
parent_texts, None, nostore_sha)
File "/usr/lib/python2.4/site-packages/bzrlib/knit.py", line 931,
in _add
access_memo = self._data.add_record(version_id, digest,
store_lines)
File "/usr/lib/python2.4/site-packages/bzrlib/knit.py", line 2020,
in add_record
size, sio = self._record_to_data(version_id, digest, lines)
File "/usr/lib/python2.4/site-packages/bzrlib/knit.py", line 1997,
in _record_to_data
["end %s\n" % version_id]))
File "/usr/lib/python2.4/site-packages/bzrlib/tuned_gzip.py", line
330, in writelines
self.write(''.join(lines))
File "/usr/lib/python2.4/site-packages/bzrlib/tuned_gzip.py", line
322, in write
self.fileobj.write( self.compress.compress(data) )
MemoryError
bzr 0.91.0 on python 2.4.4.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'commit', '-m', 'Added initial projects
and files.']
** please send this report to bazaar at lists.ubuntu.com
More information about the bazaar
mailing list