brisbane:CHKMap.iteritems() tweaks

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> On Tue, 2009-03-24 at 22:35 -0500, John Arbash Meinel wrote:
>> So there is a significant amount of overhead in
>> 'Access.get_raw_records', since we are reading 720*XMiB of compressed
>> data, even though we only care about the first few bytes out of each
>> block. And then there is a certain amount of overhead decompressing
>> the
>> extra 32KiB from each block (this seems to be trivial, as if I set the
>> extra decompress size down to 0, it only seems to save 50ms)
> 
> Why are we reading more data than needed? (I thought I had code
> [disabled] to read only the part of the compressed group needed.

We decompress only the bytes we need, but we don't have anything about
how many compressed bytes we need.

> 
>> We can also look at combining 'pack recent separately' along with
>> 'split chk by prefix'. But if the new overhead is that we now access
>> 720 blocks on disk, rather than 6 (don't forget the extra seek? to
>> read the pack header again, since this is a fresh open()), that
>> doesn't improve with 'recent'.
> 
> Well, thats where mmap may come is useful for local performance.
> 
> -Rob

Or possibly just holding the file handle open. Or perhaps our
"Transport" abstraction is starting to show its overhead.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknKH50ACgkQJdeBCYSNAANL+wCgwjrIWZd6L2Cqbnc8yEMhDwGb
hjcAnAi37R7sdeoDU9SWSjf/SPT+vFzh
=Z5at
-----END PGP SIGNATURE-----