[RFC] Blob format

[Johan Rydberg]

John Arbash Meinel <john at arbash-meinel.com> writes:

>> Absolutely.  We would be a lot better for if we had fixed-size file
>> and revision identifiers, but it is a bit too late to change that.  So
>> we probably need format with back-pointers to earlier entries in the
>> index.  
>
> That is possible, but realize it is much better to read 10k all at once,
> rather than reading 200 bytes, seeking back 200 bytes, reading another
> 200 bytes, etc.
>
> So while backpointers are good, just knowing that it is earlier than the
> current record means that you just jump back and read another 10k.

You can of course read as much as you like; the backpointers are
useful to identify where data starts as well.

> I don't know where the specific tradeoff would come, but for local disk
> access the granularity is at the very least a disk block, and more
> likely an entire stripe. So you are talking 4k -> 100+k.
> hg revlog files actually contain the data inside the index in the case
> the whole file is < 128k. So it is potentially a single disk read to get
> everything, rather than jumping around a lot.

I have not studied mercurial's revlogs in a while: Do they rewrite the
whole file if it grows above the threshold, or does it keep to the
append-only policy but put pointers to the data file for the new data?

> For remote access to an HTTP file, it depends on latency versus
> bandwidth, but my bandwidth is 160KB/s, and my latency to bazaar-vcs.org
> bounces between 100ms and 400ms.
> So at 100ms, I can download 16K, and at 400ms I can download 65K.
> So unless your back pointer is pointing further back than 16Kbytes, it
> is better to just leave it out, and read a bigger chunk.
> I think I have a moderate bandwidth system. I think we could definitely
> tune round-trip algorithms for around 20-30KB per read and do quite well.

I have about ~1MB/s with a ~35ms RTT, so yes, your bandwidth is
moderate.

Networks are a beast of their own, so probably the best way to do this
would be to hack up some small test implementations and do a set of
tests in different environments.

>> HTTP byte ranges supports negative offsets.  Not sure our readv()
>> interface does though.
>
> Unfortunately, it doesn't. But I think we should make it. file.seek()
> can do negative offsets, and I believe sftp can as well. (I'm not
> positive on that one, though).

Maybe Robey could answer this.

> And since we want to support reading just the end of index files,
> readv() should be changed. You have my blessing to do so :)

I'll take a look at it.

~j