RFC: packs and long blobs

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> So currently pack containers are length-prefixed. 
> 
> I'd like to add a method to put very large blobs into packs, so that I
> can put indices onto the end of pack containers - and indices may be
> hundreds of MB in size.
> 
> I'm thinking of requiring a file object that supports seek(0, 2); len =
> tell() in this method. Any objections/better ideas?
> 
> -Rob

Is this for reading, for writing, or both?

ISTR that there were issues with providing a "read the last 10 bytes" from a
file for all transports. I believe there was something about it being possible
in HTTP but poorly implemented by most servers? I don't remember the specifics.

We do already have Transport.stat().st_size to give the length of the file on
(I believe) all transports.

I guess I would like to understand why you think it is necessary. In general,
it isn't a great idea to read files in backwards order.

If it is because you would seek to the end, read a few bytes to find the index
start, seek to it, and then start reading the index... why not put those bytes
at the beginning. Leave a 32-byte gap to hold an ascii string of the offset
where the index starts (and possibly length). You can pad it with \0 to start
with, and then come back and write to it.

I'm not specifically opposed to the interface, I'm just not sure that it
really gives you much at this point.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFItAGMJdeBCYSNAAMRAm5vAJ9oicQtoFCOsv97tXRPKhFWTMWTWwCggaLE
TLAHhQwzlZzrERa9a1fNenw=
=jK10
-----END PGP SIGNATURE-----