committing a large (600 MB) file, p4-fast-export, fast-import
Robert Collins
robertc at robertcollins.net
Sun Feb 22 22:18:40 GMT 2009
On Sun, 2009-02-22 at 16:41 -0500, Matt McClure wrote:
> I have a Perforce repository containing a 658 MB file, that I am
> trying to mirror in Bazaar. I was able to export it with
> p4-fast-export, but I found that bzr fast-import raises MemoryError
> trying to import it. Before I go too far trying to reduce bzr
> fast-import's memory usage, I did a quick check to see if I could
> commit a 600 MB file to a Bazaar branch. No luck. It also raises
> MemoryError.
>
> I have 2 GB of memory. How much memory would Bazaar need to commit
> this file?
600MB for the lines, a dict for sequence matching, 600MB for the basis
in future commits, and a gzipped copy of the output - so 1.8GB+.
> Is there hope of reducing Bazaar's and bzr fast-import's
> memory usage?
Yes.
> ... File "C:\u\mlm\w\bzr\repo\bzr.mlm.stable\bzrlib\commit.py", line
> 867, in _record_entry
> ie, self.parent_invs, path, self.work_tree, content_summary)
> File "C:\u\mlm\w\bzr\repo\bzr.mlm.stable\bzrlib\repository.py", line
> 388, in record_entry_contents
> lines = file_obj.readlines()
> MemoryError
This is the obvious point of failure, but what is actually needed is
streaming diff generation, which is slightly harder to achieve.
-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20090223/ede033a1/attachment-0001.pgp
More information about the bazaar
mailing list