RFC: packs and long blobs

[Vincent Ladeuil]

>>>>> "robert" == Robert Collins <robertc at robertcollins.net> writes:

    robert> So currently pack containers are length-prefixed. 
    robert> I'd like to add a method to put very large blobs into packs, so that I
    robert> can put indices onto the end of pack containers - and indices may be
    robert> hundreds of MB in size.

    robert> I'm thinking of requiring a file object that supports seek(0, 2); len =
    robert> tell() in this method. Any objections/better ideas?

Objection: it's not length-prefixed anymore hence not readable as a stream.

Idea: put the indices first :)

Of course that requires knowing the size of each blob and the
number of blobs *before* writing them.

In the most degenerated scenario, you can add a 'continued' flag
to each length-prefixed chunk so that the reader become aware
that the chunk is only part of a whole.

This is dirty but I've seen it used with success to ensure
backward compatibility.

       Vincent