<html><head><meta content="text/html; charset=utf-8" http-equiv="Content-Type"></head><body><div><div style="font-family:Calibri,sans-serif;font-size:11pt">I actually saw on reddit a while ago of someone who was writing a git plug-in to handle large files, but all it really does is put the file in a 'backend' aka amazon ecs, dropbox, file system and makes sure it's there, it doesn't actually version the file with git, so not sure if that helps.<br>
<br>But I think this is a much needed feature, cause if I recall correctly, all dvcs's struggle with this, (I recall a git mailing list post where a Facebook employee was saying their repos were so large/too many binary files that it took 3 days to do a repack or something silly!), I have no idea if there is some work done on this already, but I have an interest in this as well.<br>
<br>~mark<br></div></div><hr><span style="font-family:Tahoma,sans-serif;font-size:10pt;font-weight:bold">From: </span><span style="font-family:Tahoma,sans-serif;font-size:10pt">Evan Pipho</span><br><span style="font-family:Tahoma,sans-serif;font-size:10pt;font-weight:bold">Sent: </span><span style="font-family:Tahoma,sans-serif;font-size:10pt">6/6/2012 2:56 AM</span><br>
<span style="font-family:Tahoma,sans-serif;font-size:10pt;font-weight:bold">To: </span><span style="font-family:Tahoma,sans-serif;font-size:10pt"><a href="mailto:bazaar@lists.canonical.com">bazaar@lists.canonical.com</a></span><br>
<span style="font-family:Tahoma,sans-serif;font-size:10pt;font-weight:bold">Subject: </span><span style="font-family:Tahoma,sans-serif;font-size:10pt">Extending the storage format with a plugin</span><br><br></body></html>
Hello everyone,<div><br></div><div>I am currently evaluating VCSs for a project and Bazaar is a strong contender for a few reasons.</div><div><br></div><div>1) Multiple workflows, checkouts vs branches both supported against the same central server</div>
<div>2) GUI Support, several of the contributors to this projects are non technical (artists, etc) so having fewer steps and a GUI is important.</div><div>3) Ability to use lightweight checkouts and stacked branches.</div>
<div><br></div><div>However I have a specific need that seems to not be met well in any of the current DVCSs, support for large binary files. There is a large amount of content that needs to be versioned alongside the code base, including executables, art assets, and content patch data. </div>
<div><br></div><div>The requirements are fairly simple:</div><div>1) Content is not mergeable, can only accept yours or theirs.</div><div>2) All content history must be preserved and accessible</div><div>3) Should be transparent to users of the existing bazaar tools if possible.</div>
<div>4) Content should automatically use the new storage, possibly based on a file size trigger (i.e. all files > 5MB) or if they contain a nul byte.</div><div>5) Files and content not using the new storage system would fall back to the current system which works quite well for code and text.</div>
<div><br></div><div>My initial thought is to write a plugin that uses a rolling checksum similar to rsync to break up the files and store the chunks along with metadata on how to put them back together again similar to bup (<a href="https://github.com/apenwarr/bup#readme">https://github.com/apenwarr/bup#readme</a>). When commiting the plugin would break the new file up in the same way and add any new chunks and update the metadata for the new revision.</div>
<div><br></div><div>Obviously this is still a half baked idea, but is it even possible?</div><div><br></div><div>If so, is there any documentation on this, other than <a href="http://doc.bazaar.canonical.com/plugins/en/plugin-development.html">http://doc.bazaar.canonical.com/plugins/en/plugin-development.html</a> mentioning that plugins could add new storage formats, my searches for bazaar storage format plugins came up empty.</div>
<div>Could this be developed as a fictional external source control plugin similar to <a href="http://doc.bazaar.canonical.com/migration/en/foreign/bzr-on-svn-projects.html">http://doc.bazaar.canonical.com/migration/en/foreign/bzr-on-svn-projects.html</a> ? or is there a better way?</div>
<div>Are there any existing projects, not only to handle binary files better, but anything that changes or extends bazaar storage?</div><div><br></div><div>Thanks for your time.</div><div>-Evan </div><div><br></div>