[MERGE] Developer doc: container format

[Andrew Bennetts]

Harald Meland wrote:
> [Andrew Bennetts]
> 
> > Hi all,
> >
> > This document adds a description of a basic container format to
> > doc/developers.  This was worked out with Robert on Friday.
> 
> The format appears to try being somewhat human readable, by specifying
> that newlines must be used in key positions.  So, maybe it would be a
> good idea to explicitly specify whether the length of a full-text
> record includes a final newline or not, and maybe even require an
> explicit newline to be included after the content.

I don't think a newline at the end of a full text record would help human
readability much, and you need extra complexity to deal with them.  If nothing
else they waste a little bit of space that we don't need to waste.

I think newlines internally inside headers makes sense because any byte would do
as a delimiter, but newlines in particular give you a degree of human
debuggability for free over e.g. NUL.  (And similarly I use "FT\n" as a record
marker because if the bytes are arbitrary they may as well be bytes that
slightly easier to understand when debugging that unprintables).  I don't really
care too much about human readability elsewhere in this format.  Many records
are going to contain compressed binary, so I'd expect these files to be pretty
unreadable in the raw, and I think that's fine.

-Andrew.