[MERGE] Developer doc: container format

[Andrew Bennetts]

Aaron Bentley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Andrew Bennetts wrote:
> > +This document describes the proposed container format for streaming and
> > +storing collections of data in Bazaar.  Initially this will be used for
> > +streaming revision data for incremental push/pull in the smart server for
> > +0.18, but the intentional is that this will be the basis for much more
> 
> ^^^ intention, not intentional

Fixed, thanks.

> > +It would be nice if we could combine multiple containers into a single
> > +stream by something no more expensive than concatenation (e.g. by omitting
> > +end/start marker pairs).  
> 
> ^^^ Because names are not required to be universally unique but must be
> locally unique, I don't believe this has been achieved.  Names must be
> examined and potentially rewritten when concatenating two containers.
> 
> Concatenation could be safely performed without checking names if you
> had some external guarantee that names between two containers were
> unique, but that's a subset of cases, and I'm not sure how commonly we
> will have that external guarantee.

Right.  This is not saying that it is necessarily going to produce a valid
stream, because as you say external guarantees are needed.  And obviously just
randomly taking two containers and combining them isn't necessarily going to
give you a container that's useful for anything; e.g. if both containers hold
only anonymous records whose meaning depends on their position in the container,
then such a combination would obviously produce nonsense.

All this section is say is that it's *not expensive* to do produce a container
that is a simple combination of containers, which is a useful property on its
own.  I'll clarify this.

> > +No length-prefixing of entire container
> > +---------------------------------------
> > +
> > +The overall container is not length prefixed.
> 
> For consistency, this should be "length-prefixed".

Thanks.

> > +It is acceptable to have records with no explicit name, if the expected
> > +use of them does not require them.  For example:
> > +
> > +  * a record's content could be self-describing in the context of a
> > +    particular container, or
> > +  * a record could be accessed via an index based on SHA-1, or 
> > +  * when streaming, the first record could be treated specially.
> 
> 
> Thanks for including that explanation.  It makes more sense now.
> 
> > +Specification
> > +=============
> > +
> > +This describes just a basic layer for storing simple series of "records".
> 
> I think you mean "storing a simple series"

I do.  Thanks.

-Andrew.