Merging Live and Install CDs (cloops built from .deb's)

[Paul Sladen]

On Fri, 11 Feb 2005, Matt Zimmerman wrote:
> On Fri, Feb 11, 2005 at 11:38:30PM +0000, Colin Watson wrote:
> > I believe there are also people working on a live CD installer,
> I've been thinking about this quite a bit recently.  A casper-based
> filesystem-copying install would have a number of compelling advantages:
> 
> - Somewhat increased memory requirements compared to the traditional
>   installer (though very close to those of the installed system)

It shouldn't be much more if it's done from a separate boot-option (the main
overhead being just dealing with the cloop);  unless you *wanted* the whole
stack there to get X, sound, lights, network connection---which I can see as
a considerable advantage.

> - Could not be used for upgrades

The issue here is not having the .deb's because they've already been
unpacked.

Binary Diffs    [skip down to the next section for LiveCD stuff]
============

I've been continue to think about this area in relation to binary/delta
updates (If a 10MB .deb differs by 20kB, only download the difference).

  * Mirror->Mirror (has copy of old .deb)
  * Mirror->Desktop (no old .deb)

The difference is that Desktop has all the installed files---all the
information to reconstruct the .deb;  this is exploited by 'dpkg-repack' to
build an approximation to the original .deb.

The best I have is a scatter-gather approach.  ''Here's a list of parts of
the file and where you can get them from.''  Think Bit-torrent here, once
you have grabbed the list of parts (".torrent") you can fetch a particular
piece of the jigsaw from anyone who happens to have it.

In an installed system, the 3 most likely places are:

  /usr/sbin/adduser                                    Size Datestamp MD5SUM
  /var/cache/apt/archives/adduser_0.99_all.deb         Size Datestamp MD5SUM
  http://www.example.com/debian/adduser_1.0_all.deb    Size Datestamp MD5SUM

To build the .deb to upgrade from adduser-0.99 to adduser-1.0 we first
create an empty file the size of the new 1.0.deb and then start to fill in
pieces as we find them.

Example 1.  The first place to look is the 0.99 .deb, various chunks are
copied into the new file.  Then we can look for more missing chunks, in this
case, the .deb on the mirror is known to provide 100% of the missing data
and this can be added by selective HTTP (partial-content).

Example 2.  This Matches the LiveCD/Desktop case;  the system has been
'apt-get cleaned' and there are no cached .debs to diff from.  But, each of
the installed files is able to provide a small piece of the puzzle (or at
least gziped, they can).  Again any missing chunks required can be fetched
from a Mirror, or another chunk-store (eg. specially created file that just
happens to contains the missing bits...).

One disadvantage of the chunk approach when talking about a LiveCD, as a
upgrade source, is that the .debs have to be built---even if that is only in
a RAM disk just before use.

[...notes about various hacks (hard-linking of sectors) to get file-system
visible .deb's from the cloop-fs or the ISO...deleted]

LiveCD cloop built from Deb's
=============================

The brainwave Eureka moment--it is alot easier to build the cloop fs from
the .deb's than it is to uncompress (gunzip), install and recompress (gzip)
everything and then try to reconstruct the .deb's.

If you've never looked inside a .deb, in a abstracted raw sense it is:

  132 byte header (!<arch>, "debian-binary...", "version", "data.tar.gz...")
  Filesystem tarball:
    {
      gzip'ed 512 tar header
      gzip'ed FILE                    <--- IMPORTANT BIT
      gzip'ed padding to next 512
    } * X files
  Control tarball:
    { Same as above } * Y files
  XX byte footer (more recently containing _gpg* signatures)

Two separate gzip streams concatrenated together produces the same output as
one gzip stream.  If the gzip stream for the actual FILE is sync'ed once
every 64kB, (They effectively already are if --resyncable is used), pointers
to each (64kB) block would immediately be useable as cloop blocks.

It's not so much about teaching 'create_compressed_fs' about .deb's as
teaching it to read Tar files.

Because a LiveCD is a snapshot after install, once the initial filesystem
image is built, it would need to be mounted and ''executed'' with all the
changes introduced by the configuration control files tagged on at the end.
If you like, ''appending another tarball to the end'';  this bit would be
tricky since otherwise this technique lends itself to producing a read-only
filesystem---the difference to cloop is that the 'read-only' part is
performed at post-install.  This could be done in a similar way to the
current LiveCD dm-layer use to provide copy-on-write support into RAM.

The actual hard-linking of sectors between the cloop-image and the main
iso9660 filesystem would be done by mkisofs/jigdo both of which currently
know about this sort of thing to a point, but would need more a bit more
loving on each part.

For somebody wanting to modify the LiveCD image without retaining the
'hard-links' to the .deb's, they would be able to copy the cloop file,
uncompress it exactly as they currently do with 'extract_compressed_fs'
and mount it.  It should be no problem to create backwards compatible cloop,
in the same way that jigdo creates a normal ISO despite just building it
from .debs.

Suddenly the idea of having a combined Live/Install CD doesn't sound quite
so ugly after all...

	-Paul
-- 
Is there no safe way to travel?  London, GB