Extending the storage format with a plugin

[Chris Hecker]

This comes up every few months.  I believe pretty strongly that the
hacky solutions that hg and git use are not the right thing, but it
appears there's nobody interested in doing the right thing, so maybe
they're better than nothing.  I don't know.  You can search the list
archives for plenty of threads on this.

Chris

On 2012/06/06 02:55, Evan Pipho wrote:
> Hello everyone,
> 
> I am currently evaluating VCSs for a project and Bazaar is a strong
> contender for a few reasons.
> 
> 1) Multiple workflows, checkouts vs branches both supported against the
> same central server
> 2) GUI Support, several of the contributors to this projects are non
> technical (artists, etc) so having fewer steps and a GUI is important.
> 3) Ability to use lightweight checkouts and stacked branches.
> 
> However I have a specific need that seems to not be met well in any of
> the current DVCSs, support for large binary files.  There is a large
> amount of content that needs to be versioned alongside the code base,
> including executables, art assets, and content patch data. 
> 
> The requirements are fairly simple:
> 1) Content is not mergeable, can only accept yours or theirs.
> 2) All content history must be preserved and accessible
> 3) Should be transparent to users of the existing bazaar tools if possible.
> 4) Content should automatically use the new storage, possibly based on a
> file size trigger (i.e. all files > 5MB) or if they contain a nul byte.
> 5) Files and content not using the new storage system would fall back to
> the current system which works quite well for code and text.
> 
> My initial thought is to write a plugin that uses a rolling checksum
> similar to rsync to break up the files and store the chunks along with
> metadata on how to put them back together again similar to bup
> (https://github.com/apenwarr/bup#readme).  When commiting the plugin
> would break the new file up in the same way and add any new chunks and
> update the metadata for the new revision.
> 
> Obviously this is still a half baked idea, but is it even possible?
> 
> If so, is there any documentation on this, other
> than http://doc.bazaar.canonical.com/plugins/en/plugin-development.html mentioning
> that plugins could add new storage formats, my searches for bazaar
> storage format plugins came up empty.
> Could this be developed as a fictional external source control plugin
> similar
> to http://doc.bazaar.canonical.com/migration/en/foreign/bzr-on-svn-projects.html ?
> or is there a better way?
> Are there any existing projects, not only to handle binary files better,
> but anything that changes or extends bazaar storage?
> 
> Thanks for your time.
> -Evan 
>