Out of Memory a bridge too far
Jason Earl
jearl at notengoamigos.org
Sat Nov 12 09:14:09 UTC 2011
On Sat, Nov 12 2011, Martin Pool wrote:
>> Out of curiosity what would an acceptable solution for this sort of
>> thing look like to the Bazaar development team? I don't really want to
>> switch to anything else, and so I can't help but be curious as to what
>> it would take to make bzr work with large files.
>
> There are two main things you can do:
> One is to make and publish a 64-bit Windows Python build. Just
> getting a 2.4.3 would be great, so you don't need to commit to doing
> it for every release though of course that would be nice. This
> doesn't need any particular Python skills, just a certain amount of
> work to go through the manual environment setup and then the automated
> scripts that are in bzr-windows-installers. If you start on this and
> run into errors or confusion I'm sure people would be happy to help.
> That should at least let Bazaar use all of the memory on your machine
> rather than being limited to ~2GB or maybe less.
>
> I can really understand your frustration when it runs out of memory,
> but I find it a bit frustrating people complain about this but won't
> step up to make a Windows 64 bit build.
>
> The other avenue is to actually just fix things that do use more
> memory than they should. To do this you typically need to run bzr
> with -Dmem_profile, run a command, look into where it's using memory,
> and basically fix it. With this too, if you show interest at least to
> getting a dump and saying "what do I do now?" people will help you
> with it. John did a lot of work on this in 2.4 (and as it happens
> I've just been deploying that into Launchpad through
> <http://blog.launchpad.net/general/daily-builds-huge-trees>.) He has
> a worked example of how to use the Meliae profiler in
> <http://jam-bazaar.blogspot.com/2010/08/step-by-step-meliae.html>.
>
> The other thing Liam mentioned is to warn people before they commit
> large files, so they don't get into this sticky situation, and Shannon
> Weyrick recently contributed a patch to do that in
> <https://code.launchpad.net/~weyrick/bzr/54624-warn-on-large-files/+merge/70691>.
>
> m
Thanks for answering, as the reason I asked is that I am at least
somewhat interested in actually *doing* something.
However, I don't have Windows-specific problems with large files. I am
a GNU/Linux user (Ubuntu actually), and I am working with a mixture of
source code and binary files up to about 500M or so. My most
problematic project is comprised of a mixture of LaTeX and image files.
I will inevitably want to have larger files as well.
What I am actually looking for (I believe) is something along the lines
of the new largefile support in Mercurial. I want to be able to say
"this is a big binary file, treat it differently," and call it a day.
In a perfect world I would not only like to write something that I could
use, but something that would be accepted back into Bazaar.
Unfortunately, I have no idea if the Bazaar developers would even be
interested in such a beast, much less how hard something like that would
be to write.
I suppose if it were easy it would probably already be done.
Jason
More information about the bazaar
mailing list