Modular LiveCD?

[Lutzer]

Probably you could get more discussion at a developer mailinglist.

John Richard Moser schrieb:
> (I am not on sounder, CC me things)
> 
> Does anyone think it's worth specing out a plan for modular LiveCDs in
> the vein of Slax and Morphix for Dapper+9999?  I was thinking something
> to allow flexible LiveCD creation and customization, with a front-end
> tool to inject standard base system elements and add-on modules using
> UnionFS to combine them.
> 
> NOTE:  here we are assuming a hypothetical tool called "LiveAuthor" that
> creates or 'authors' LiveCDs
> 
> I have two major goals with this proposal:
> 
>  1.  Make LiveCD remastering quick and painless, to the point that the
> most work needed by the end user is typing commands in a shell (the CD
> is built automatically)
>  2.  Modularize the LiveCD so that most of the components are built once
> per version per architecture; in other words, if KDE is used on Edubuntu
> and Kubuntu, then KDE for IA-32 should be built/installed/configured one
> time for both of them.
> 
> 
> The basic plan I'm thinking is as such:
> 
>  - Ubuntu developers create the skeletal LiveCD, a non-functional LiveCD
> which basically can't even boot as is, but is the facilitator for
> created LiveCDs.  This is called the "Bootstrap."
>  - The Bootstrap is seeded with "Boot Modules," archives containing
> kernels, modules, and bootloader configuration files for either Syslinux
> or the grub that Morphix uses (check out PHLAK or Gnoppix).
>  - The Bootstrap is seeded with "Base Modules," typically only 1, which
> includes a basic Linux installation.
>  - The Bootstrap is seeded with "Extension Modules," typically many,
> which include things such as desktop environments (XFCE, KDE), X11,
> tools (Gimp, security tools, etc), and the like.
>  - During the bootstrap, steps may need to be taken based on Base and
> Extension modules.  These can be scripted in Bootstrap Modules
> 
> Specific details for the Boot Modules:
> 
>  - They supply CDROM/boot/<kernel>/vmlinuz kernel image
>  - They supply CDROM/boot/<kernel>/config configuration
>  - They supply CDROM/boot/<kernel>/modules.squash kernel modules
> squashfs image
>  - The bootloader configuration file is generated to source each
> configuration
>  - The initrd is generated at ISO build time
>   - In the future, the kernel should be able to have a second "initrd"
> loaded, which would be the cramfs image, to reduce redundant data (i.e.
> kernel modules AND the contents of initrd)
>  - /CDROM/boot/<kernel>/modules.squash will be mounted to final
> /lib/modules/<kernel>
> 
> Specific details about the base modules:
> 
>  - They contain the basic boot environment
>  - Probably should only contain a command line boot
>  - Mounted as our final / to be pivot_root target
>  - Should be modeled as a real base system not accounting for running
> from LiveCD
> 
> Specific details about the Extension Modules:
> 
>  - Contain additional utilities such as X11, command line tools, GUI
> tools, graphical environments (KDE, XFCE)
>  - Are UnionFS mounted on top of our final / before pivot_root
>  - At ISO creation, Extension Modules can be set to depend on other
> extension modules, i.e. XFCE on GTK+2 and X11.
> 
> Specific details about the Boot Modules creation process:
> 
>  - Ubuntu developers release the kernel image package
>  - Boot Module creator loads the .deb into LiveAuthor
>  - LiveAuthor extracts the kernel and modules
>  - LiveAuthor produces an initrd, bootloader configuration, and
> modules.squash
>  - modules.squash contains the contents of /lib/modules/<kernel> such
> that it can be mounted directly at /lib/modules/<kernel>
>  - LiveAuthor places the output in a gzip/bzip2 tarball with directory
> structure:
>   - '/<kernel>' where <kernel> is `uname -r` for that kernel
>   - '/<kernel>'/{config,vmlinuz,modules.cram}
> 
> Specific details about the Base Module creation process:
> 
>  - Base Module creator uses LiveAuthor to start a Base Module
>  - LiveAuthor either debbootstraps a project directory or selects an
> existing one
>  - LiveAuthor spawns a terminal with a shell chroot()ed to the project
> directory
>  - When the shell is exited, LiveAuthor allows the Base Module creator
> to generate a SquashFS image from the directory.
>  - LiveAuthor could be instructed to run certain Bootstrap Modules for
> certain Base Modules
> 
> Specific details about the Extension Module creation process:
> 
>  - Extension Module creator uses LiveAuthor to start an Extension Module
>  - LiveAuthor makes the Extension Module creator select Base Modules and
> Extension Modules to start with
>  - LiveAuthor mounts and unionfs-mounts these modules together, then
> unionfs-mounts a project directory on top of this stack
>  - LiveAuthor spawns a terminal with a shell chroot()ed into the mountpoint
>  - Upon exiting the shell, LiveAuthor allows the Extension Module
> creator to create a SquashFS of the project directory, including all
> white-out files, such that unionfs mounting the resulting image on top
> of the underlying stack will re-create the current state.
>  - Extension Modules could be configured to depend on Bootstrap Modules;
> for example, X11 may depend on X11.bootstrap to configure X.
> 
> Specific details about the creation process:
> 
>  - Ubuntu developers release the Bootstrap, kernel image, and Ubuntu
> packages
>  - Boot Module is created via the kernel package
>  - Base Module is created via putting in the legwork to generate one
> (ubuntu-base?)
>  - Extension modules are created (ubuntu-standard? ubuntu-desktop?
> Xubuntu-desktop)
>  - LiveAuthor is used to merge the Bootstrap, Kernel Module, Base
> Modules, and Extension Modules into an ISO image
> 
> General comments on the creation process:
> 
>  - "Remastering" the CD would involve having LiveAuthor decompose the
> CD, which is trivial; or simply snagging the Bootstrap, Kernel Package,
> and Base and Extension Modules and working from there
>  - Additional Extension Modules can easily be added to the CD--
> LiveAuthor would simply merge the Bootstrap with Kernel, Base, and
> Extended Modules again, with whatever new ones.
>  - Creating new Extension Modules would be done by having LiveAuthor
> reconstruct the mounted environment from an existing LiveCD and then
> mount another directory UnionFS atop that, as normal.
> 
> Specific details about the LiveCD boot process:
> 
>  - The LiveCD would boot from a configuration file sourcing or generated
> from each individual Kernel Module.
>  - The kernel would boot, running the initrd to gain CD access and then
> starting the Bootstrap.
>  - The Bootstrap would use a script generated at ISO creation time by
> LiveAuthor to decide which modules to use based on kernel/bootloader
> command line instructions and/or user input.
>  - The selected Base Module would be mounted at /target
>  - Each Extension Module would be mounted at /target via UnionFS
>  - Pre-tmpfs Bootstrap Modules are run here
>  - A tmpfs is bind mounted over /target via UnionFS, unless a Pre-tmpfs
> Bootstrap Module DISABLED this.
>  - At this point, certain known files are handled, i.e. dpkg database
> _changes_ between modules (in case packages are installed or removed)
> are merged
>  - Post-tmpfs Bootstrap Modules are run to configure networking, X11,
> etc. and write the changes into /target
>  - /target is set to / via pivot_root
>  - 'exec /sbin/init --init' is called to replace the Bootstrap initrc
> with real init
> 
> I believe that this sort of design has several advantages.  To be brief:
> 
>  - Creating a LiveCD involves adding files to a tree and running a
> program to automagically spit out the CD.
>  - Creating Base Modules is a matter of clicking a button and being
> dropped into a shell
>  - Extension Modules can allow maintenance to be broken down.  If a
> change is made and something breaks, it's likely either in the base
> module or the extension.
>  - Extension Modules can be created based on existing LiveCDs and
> released on the net.
>  - CDs can easily be extended by downloading or creating new Extension
> Modules and merging them.
>  - Combo DVDs can be created by taking the set of extension modules used
> in various LiveCDs and creating a new image; this could for example
> quickly create an XFCE/Gnome/KDE DVD from Ubuntu, Kubuntu, and Xubuntu's
> Extension Modules.
>  - Minor modifications like rebranding, theming, or wallpapering can be
> made quickly and easily by creating a new extension module, which in
> turn involves just hacking around in a shell and saving the changes.
>  - Some of the heavy lifting can be reduced.  Building Kubuntu, Xubuntu,
> Ubuntu, and Edubuntu LiveCDs would involve building the Bootstrap;
> building the Kernel Modules; building the Base Modules; building the X11
> modules; building the GTK+ and Qt modules; building the KDE, GNOME, and
> XFCE modules; and building the Edubuntu module with additional software.
>  Each CD would be generated using the same Bootstrap, Base Module, and
> X11/GTK/Qt Extension Module.  Edubuntu would use the same KDE or GNOME
> extension module as Kubuntu/Ubuntu.  This means each of these packages
> is built ONE TIME per architecture.
>  - Testing of changes can be done quickly and painlessly.  Changes to an
> Extension Module can be done by making those changes in a new Extension
> Module and then testing them before merging with the original.  Changes
> to the boot process can also be done with Bootstrap Modules; for
> examples, see the next section.
> 
> There are some interesting thoughts here on Bootstrap Modules that could
> be taken out somewhere in the future:
> 
>  - Bootstrap Modules can be used to prevent mounting a tmpfs over
> /target and instead unionfs with a USB stick or external hard drive.  By
> doing this, changes become persistent.
>  - A Bootstrap Module could be used to add users to the system, disable
> GDM auto-login, and bind mount a /home directory from a USB stick inside
> /home.  This would facilitate a persistent /home.
>  - As with the above, a roaming /home could be done by copying from SMB,
> NFS, or USB and periodically syncing back to the source of the /home
> directory.
>  - While we are loading things from stuff plugged into the machine, it's
> possible to locate additional Extension Modules and Bootstrap Modules
> and load those too, without altering the CD.
>  - All of this USB stick stuff would have to look for a file like
> /media/usbdrive/ubuntu-control/config or something
> 
> Final thoughts:
> 
>  - LiveAuthor would probably benefit heavily from logging actual shell
> commands entered.  This would allow upgrading or rebuilding modules for
> breaking changes in other Base/Extension Modules to be nearly automated.
>  - Copying external files into a module would be a common action.
> Handling this could be done via either a built-in file browser or
> integration with Nautilus.
> 
> Any questions?  Comments?
> --
> All content of all messages exchanged herein are left in the
> Public Domain, unless otherwise explicitly stated.
> 
>     Creative brains are a valuable, limited resource. They shouldn't be
>     wasted on re-inventing the wheel when there are so many fascinating
>     new problems waiting out there.
>                                                  -- Eric Steven Raymond
> 
>     We will enslave their women, eat their children and rape their
>     cattle!
>                                      -- Evil alien overlord from Blasto