Zsync for Packages files

[Lars Wirzenius]

I did a test to see how much benefit zsync could give to downloading
Packages files.

I did snapshots of karmic Packages files every four hours for a week,
and simulated downloading the file with zsync. I measured the update
from each file to the next and checked how much bandwidth was saved.

When the Packages file is compressed with "gzip --rsyncable", I get a
minimum of 75% reused, maximum 99.9% reused, and a median of 91.9%
reused.

It would seem to me that Packages files are, indeed, low-hanging fruit.

If we can provide un-compressed Packages files, results are even better:
minimum 95.5%, maximum 99.9%, median 99.2% re-used data.

Obviously, these results are relevant only for people who update really
often, but I expect results for things like -security to be very
similar, since it changes fairly little. Results should be worse for
people following a development release who don't update regularly, since
they will have a bigger delta.

The .zsync files (using default block size) are about 0.3% of the size
of the original files, so that should be insignificant. We also don't
have enough Packages files to worry about the number of new files we
would add.

I have not compared zsync with pdiffs. Ubuntu does not use pdiffs
because it generates Packages files every hour, and it seems that ruins
the usability of pdiffs.