brisbane:CHKMap.iteritems() tweaks

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Arbash Meinel wrote:
...

> 
> I did propose changing the chk grouping to be pure
> 
> R1 R2 R3 R4 R5 R6
> A1 A2 A3 A4 A5 A6
> B1 B2 B3 B4 B5 B6
> C1 C2 C3 C4 C5
> 
> But to do that easily, I would need to create more streams. Also, when I
> was trying to tweak this, the initial results showed things being worse
> for file texts. So it may need some other work to try to tweak the
> compress code to know what it is compressing, so that it know it is
> dealing with file texts, which would have a different expected behavior
> than chk texts, etc.
> 

...

> I'll poke at it a bit. But anyway, the same thing holds true for the
> file texts. Because it was one of the ways that we got better
> compression. (I'm guessing it is cross-file compression of stuff like
> copyright headers.)
> 
> John
> =:->

So splitting things in this fashion, I see :
 time bzr ls launchpad-packed
 7.28s

 time bzr ls launchpad-split
 3.25s

 time bzr ls launchpad-first-100
 2.49s


'bzr ls' changes to read in 723 blocks up from 6 blocks. (which is
expected, because now all chk pages that start with a different prefix
end up in a different group.)

Which ends up being 723 calls to 'make_readv_reader' up from 6, and 1446
'_iter_records' (probably because we, yet-again, read the header of the
pack file).

This shows up as 846ms of '_get_block' time, up from 33ms. Which is
faster, per-block, but ultimately a lot slower :).

get_bytes_as is up from 110ms to 283ms (both for 720 texts). Which seems
to be primarily from the extra calls to zlib.decompressobj.decompress
(723 up from 9 calls). This may be because the code is currently tuned
to extract 32KiB extra from all extract calls. And 720*32KiB = 22MiB
that we are decompressing.

So there is a significant amount of overhead in
'Access.get_raw_records', since we are reading 720*XMiB of compressed
data, even though we only care about the first few bytes out of each
block. And then there is a certain amount of overhead decompressing the
extra 32KiB from each block (this seems to be trivial, as if I set the
extra decompress size down to 0, it only seems to save 50ms)

This is still a lot better than the old packing. The disk space cost is
fairly low:
127468  launchpad-chk255big-packed
127772  launchpad-chk255big-chk-split
127924  launchpad-chk255big-packed-fresh-100-notext

Consider that the old form was accessing 152 unique blocks, so had less
overhead in that aspect, but did have 3.9s spent in zlib.decompress()

We can also look at combining 'pack recent separately' along with 'split
chk by prefix'. But if the new overhead is that we now access 720 blocks
on disk, rather than 6 (don't forget the extra seek? to read the pack
header again, since this is a fresh open()), that doesn't improve with
'recent'.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknJpo8ACgkQJdeBCYSNAANeBgCgipEn/+/ZZOyt2jup9+VDuRnF
Y34AoJN7XEPkn/XGFGTXMLLvJxXSX+4F
=oN6I
-----END PGP SIGNATURE-----