committing a large (600 MB) file, p4-fast-export, fast-import

[Ian Clatworthy]

Matt McClure wrote:
> I have a Perforce repository containing a 658 MB file, that I am
> trying to mirror in Bazaar.  I was able to export it with
> p4-fast-export, but I found that bzr fast-import raises MemoryError
> trying to import it.  Before I go too far trying to reduce bzr
> fast-import's memory usage, I did a quick check to see if I could
> commit a 600 MB file to a Bazaar branch.  No luck.  It also raises
> MemoryError.

Hi Matt,

Can I ask how important that file is? Often files of that size are
generated rather than raw data. If you don't really need it,
bzr fast-import-filter might be useful for taking it out of the
data stream so you can at least work with everything else.

> I have 2 GB of memory.  How much memory would Bazaar need to commit
> this file?  Is there hope of reducing Bazaar's and bzr fast-import's
> memory usage?

As Robert indicated, we want to reduce bzr's memory footprint (but
it's not top of the list just yet). I don't expect fast-import to be
the primary issue here. It can certainly be improved but I'm hacking
on it a lot currently (in order to improve its performance on the
brisbane-core branch) so I'd suggest holding off on trying to
improve it's memory usage until after I'm done on that.

Ian C.