Rsyncable LiveCD overview

[Matt Zimmerman]

On Sat, Jan 15, 2005 at 09:45:48PM +0000, Paul Sladen wrote:

> Here, the decompressor has not changed at all, but the compressor has been
> told to synchoronize the stream every 4 input bytes.  This has made a larger
> compressed version (about 0.07% larger in the real-world), but rsync can now
> see matches before and after the New Data in the middle.
> 
> This idea is already built into 'gzip' and can be enabled with the
> '--rsyncable' argument.  In Ubuntu and Debian, all the '.deb's are built
> this way to reduce rsync-traffic.  The install CD ISO images are built by
> Jigdo by concatranating (connecting together) all these '.deb's and adding
> the filesystem magic around them.  This is why the install CD syncs fine
> between revisions.

The reason why install CDs rsync well is that they contain large swaths of
unchanged data (i.e., the debs), not because the contents of the debs are
rsyncable (they aren't, as far as I know).

> Since there are no changes to the decompressor, only zlib itself needs
> patching for the compression stage (and only the copy linked to the cloop
> tools);  the 'create_compressed_fs' command in 'cloop-utils' can then use
> this without any modifications.  Matt added that the compressor in use for
> Ubuntu images is really the newer 'advfs' installed with a different name.
>
> A Hungarian distribution hit something similar (with apt-rsync) a couple of
> years ago, ported the original '--rsyncable' patch and added a simple hack
> to test an environment variable to transparently enable the sychronisation
> code without the application needing to know or be recompiled:
> 
>   http://lists.debian.org/debian-devel/2003/07/msg00462.html
> 
> Patches  2a, 2b & 3  are the patches to pull in, the last of which is the
> environment variable patch, but I'm not sure if that would get accepted
> upstream, whereas adding an extra flag to the 'level' parameter of
> zlib's compress2() might be more paletable.

It certainly would be nice to be able to do this without including a private
copy of zlib in cloop-utils; we learned from the last zlib security
vulnerability just how much work it can create to duplicate this code all
over the place.

> Lamont says he's going to try and work on it (I guess not having broadband
> is a motivator!)

Sounds great!  We can try it out with a hacked-up copy as a proof of
concept.

This has come up many times before, and I'm eager to hear what the word is
from upstream on integrating this functionality in a sane way.  It's
incredibly useful.

-- 
 - mdz