RFC: packs and long blobs

[Robert Collins]

On Tue, 2008-08-26 at 08:13 -0500, John Arbash Meinel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Robert Collins wrote:
> > So currently pack containers are length-prefixed. 
> > 
> > I'd like to add a method to put very large blobs into packs, so that I
> > can put indices onto the end of pack containers - and indices may be
> > hundreds of MB in size.
> > 
> > I'm thinking of requiring a file object that supports seek(0, 2); len =
> > tell() in this method. Any objections/better ideas?
> > 
> > -Rob
> 
> Is this for reading, for writing, or both?

Writing.

> ISTR that there were issues with providing a "read the last 10 bytes" from a
> file for all transports. I believe there was something about it being possible
> in HTTP but poorly implemented by most servers? I don't remember the specifics.

I'm not proposing to read the last 10 bytes, as I said to vila, the pack
disk format will be unchanged. Its *purely* about allowing writing to
not need a fully-buffered blob.

> We do already have Transport.stat().st_size to give the length of the file on
> (I believe) all transports.
> 
> I guess I would like to understand why you think it is necessary. In general,
> it isn't a great idea to read files in backwards order.

Nothing to do with reading.

> If it is because you would seek to the end, read a few bytes to find the index
> start, seek to it, and then start reading the index... why not put those bytes
> at the beginning. Leave a 32-byte gap to hold an ascii string of the offset
> where the index starts (and possibly length). You can pad it with \0 to start
> with, and then come back and write to it.

Huh? I have no idea what you are saying here.

> I'm not specifically opposed to the interface, I'm just not sure that it
> really gives you much at this point.

It would shrink memory usage for a spike I've done by hundreds of
megabytes.

-Rob
-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20080826/cba995bf/attachment.pgp