committing a large (600 MB) file, p4-fast-export, fast-import

[Robert Collins]

On Sun, 2009-02-22 at 16:41 -0500, Matt McClure wrote:
> I have a Perforce repository containing a 658 MB file, that I am
> trying to mirror in Bazaar.  I was able to export it with
> p4-fast-export, but I found that bzr fast-import raises MemoryError
> trying to import it.  Before I go too far trying to reduce bzr
> fast-import's memory usage, I did a quick check to see if I could
> commit a 600 MB file to a Bazaar branch.  No luck.  It also raises
> MemoryError.
> 
> I have 2 GB of memory.  How much memory would Bazaar need to commit
> this file? 

600MB for the lines, a dict for sequence matching, 600MB for the basis
in future commits, and a gzipped copy of the output - so 1.8GB+.

>  Is there hope of reducing Bazaar's and bzr fast-import's
> memory usage?

Yes.

> ...  File "C:\u\mlm\w\bzr\repo\bzr.mlm.stable\bzrlib\commit.py", line
> 867, in _record_entry
>     ie, self.parent_invs, path, self.work_tree, content_summary)
>   File "C:\u\mlm\w\bzr\repo\bzr.mlm.stable\bzrlib\repository.py", line
> 388, in record_entry_contents
>     lines = file_obj.readlines()
> MemoryError

This is the obvious point of failure, but what is actually needed is
streaming diff generation, which is slightly harder to achieve.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20090223/ede033a1/attachment-0001.pgp