[Ubuntu-phone] Image build infrastructure revamp status

Sun May 11 22:46:04 UTC 2014

On Sun, May 11, 2014 at 11:08:31PM +0100, Colin Watson wrote:
> [This is a Foundations/CI project; I haven't posted a status update on
> it before, but a few people have asked about it in private mail, so it
> might be helpful to have a public update to refer to.]
> 
> We've been working on reorganising the infrastructure for the live
> filesystem parts of our image builds.  At the moment there's a single
> dedicated builder for each architecture that the central image build
> system (lp:ubuntu-cdimage, running on nusakan.canonical.com) can ask to
> produce various live filesystems; ubuntu-cdimage then fetches the
> results from it, performs some final polishing steps (for ISO images,
> this involves building a pool of .debs among other things, so is a
> fairly complex procedure requiring debian-cd and a local mirror; for
> phone images it's relatively simple), and publishes the output on
> cdimage.ubuntu.com.  In the case of phone images, system-image then
> steps in to turn these into the system-image format and compute deltas
> etc.
> 
> This is serviceable for what we're doing with it right now, but there
> are a few motivations for improving things which have been on the
> wishlist for some time:
> 
>  * It's not at all horizontally scalable.  We notice this most around
>    milestone releases when we try to do lots of image builds at once,
>    and we can only do one at a time per architecture.  This is basically
>    why respin cycles for milestones take eight hours or whatever
>    ridiculous amount it is.
> 
>  * We can't sensibly extend this to experimental builds.
> 
>  * To a first approximation, ubuntu-cdimage has only been deployed once
>    (although I rewrote it a while back, and one of the goals of the
>    rewrite was to make deployment easier).
> 
> We discussed this at the release engineering sprint last year, and the
> general consensus was that we should move live filesystem building into
> Launchpad.  This has a number of advantages: both build farms are
> normally well within capacity except for occasional bursts that tend to
> be at complementary times; we would be able to run many more image
> builds at once and thus drastically reduce the time required for a full
> respin; it would be much easier to monitor builds in progress; builds
> will be cancellable without sysadmin intervention; and we'll get one or
> two extra builders on most architectures by merging the build farms, at
> least where the hardware in use is new enough to make that worthwhile.
> 
> I've been working on this on and off for a while, and essentially
> full-time since 14.04 released.  The current status is as follows:
> 
>  * The launchpad-buildd package in production already has most of the
>    code required to build live filesystems; this was done towards the
>    end of last year.  More recent testing uncovered a few small but
>    critical problems with this code with various real-world image types
>    (buggy handling of symlinks and of duplicate files in build output);
>    one fix is in trunk and the other is awaiting review.
> 
>  * The bulk of the required changes to Launchpad itself have been
>    through one review pass and are awaiting re-review.  The required
>    extensions to the web UI are still only on my local disk, as I have a
>    few more tests to write first, but I expect to finish that in the
>    first half of the coming week.
> 
>  * I have a local installation that is capable of building full live
>    filesystem images, tested with both Ubuntu Desktop and Ubuntu Touch
>    (i386).
> 
>  * I have a local ubuntu-cdimage branch that can request live filesystem
>    builds from a local Launchpad instance and wait for them to complete.
>    It cannot yet download the results.
> 
> There's still a fair bit to do, but my aim is to have this landed and in
> production in time for the core sprint in Malta in the last week of May.
> 
> The engineers developing the CI Airline need to be able to build live
> filesystems against PPAs as a component of their work, and so I've been
> in discussions with them to make sure that we will be able to use the
> same software for this, and where possible the same infrastructure,
> rather than duplicating work.  In anticipation of this requirement, the
> database modelling for live filesystems in Launchpad includes the
> facility to associate a build with a PPA, although the pieces required
> to pass that through to the sources.list used to construct the live
> filesystem aren't quite in place yet.  The CI Airline will need its own
> cut-down ubuntu-cdimage and system-image instances, and I'll be working
> with Paul Larson early this week to get the ubuntu-cdimage part of that
> up and running; phone image builds are simple enough from this point of
> view that I don't expect that to be difficult.  Although I can't speak
> for system-image, I think we are well on course to have all the rest of
> this either done before Malta or easy to finish off there.

I'm mostly off until Malta, only around this Friday and Monday-Tuesday
of next week (16, 19 and 20th of May) so if you have cdimage setup by
then, it's possible I can help setting things up for system-image
builds, if you need more time, then it's just a matter of sitting
together with Paul in Malta and getting the remaining bits sorted out (I
expect we'll need some minor tweaks to the code but nothing too scary).

> 
> Cheers,
> 
> -- 
> Colin Watson                                       [cjwatson at ubuntu.com]
> 
> -- 
> Mailing list: https://launchpad.net/~ubuntu-phone
> Post to     : ubuntu-phone at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~ubuntu-phone
> More help   : https://help.launchpad.net/ListHelp

-- 
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20140511/1f541224/attachment.pgp>