Image build infrastructure revamp status

[Colin Watson]

[This is a Foundations/CI project; I haven't posted a status update on
it before, but a few people have asked about it in private mail, so it
might be helpful to have a public update to refer to.]

We've been working on reorganising the infrastructure for the live
filesystem parts of our image builds.  At the moment there's a single
dedicated builder for each architecture that the central image build
system (lp:ubuntu-cdimage, running on nusakan.canonical.com) can ask to
produce various live filesystems; ubuntu-cdimage then fetches the
results from it, performs some final polishing steps (for ISO images,
this involves building a pool of .debs among other things, so is a
fairly complex procedure requiring debian-cd and a local mirror; for
phone images it's relatively simple), and publishes the output on
cdimage.ubuntu.com.  In the case of phone images, system-image then
steps in to turn these into the system-image format and compute deltas
etc.

This is serviceable for what we're doing with it right now, but there
are a few motivations for improving things which have been on the
wishlist for some time:

 * It's not at all horizontally scalable.  We notice this most around
   milestone releases when we try to do lots of image builds at once,
   and we can only do one at a time per architecture.  This is basically
   why respin cycles for milestones take eight hours or whatever
   ridiculous amount it is.

 * We can't sensibly extend this to experimental builds.

 * To a first approximation, ubuntu-cdimage has only been deployed once
   (although I rewrote it a while back, and one of the goals of the
   rewrite was to make deployment easier).

We discussed this at the release engineering sprint last year, and the
general consensus was that we should move live filesystem building into
Launchpad.  This has a number of advantages: both build farms are
normally well within capacity except for occasional bursts that tend to
be at complementary times; we would be able to run many more image
builds at once and thus drastically reduce the time required for a full
respin; it would be much easier to monitor builds in progress; builds
will be cancellable without sysadmin intervention; and we'll get one or
two extra builders on most architectures by merging the build farms, at
least where the hardware in use is new enough to make that worthwhile.

I've been working on this on and off for a while, and essentially
full-time since 14.04 released.  The current status is as follows:

 * The launchpad-buildd package in production already has most of the
   code required to build live filesystems; this was done towards the
   end of last year.  More recent testing uncovered a few small but
   critical problems with this code with various real-world image types
   (buggy handling of symlinks and of duplicate files in build output);
   one fix is in trunk and the other is awaiting review.

 * The bulk of the required changes to Launchpad itself have been
   through one review pass and are awaiting re-review.  The required
   extensions to the web UI are still only on my local disk, as I have a
   few more tests to write first, but I expect to finish that in the
   first half of the coming week.

 * I have a local installation that is capable of building full live
   filesystem images, tested with both Ubuntu Desktop and Ubuntu Touch
   (i386).

 * I have a local ubuntu-cdimage branch that can request live filesystem
   builds from a local Launchpad instance and wait for them to complete.
   It cannot yet download the results.

There's still a fair bit to do, but my aim is to have this landed and in
production in time for the core sprint in Malta in the last week of May.

The engineers developing the CI Airline need to be able to build live
filesystems against PPAs as a component of their work, and so I've been
in discussions with them to make sure that we will be able to use the
same software for this, and where possible the same infrastructure,
rather than duplicating work.  In anticipation of this requirement, the
database modelling for live filesystems in Launchpad includes the
facility to associate a build with a PPA, although the pieces required
to pass that through to the sources.list used to construct the live
filesystem aren't quite in place yet.  The CI Airline will need its own
cut-down ubuntu-cdimage and system-image instances, and I'll be working
with Paul Larson early this week to get the ubuntu-cdimage part of that
up and running; phone image builds are simple enough from this point of
view that I don't expect that to be difficult.  Although I can't speak
for system-image, I think we are well on course to have all the rest of
this either done before Malta or easy to finish off there.

Cheers,

-- 
Colin Watson                                       [cjwatson at ubuntu.com]