That upstart Upstart

[Colin Watson]

On Mon, Jun 29, 2009 at 04:11:53PM -0500, Patrick Goetz wrote:
> Everyone knows the canonical (no pun intended) unix interview question, 
> namely
> 
>   Q: how many processes does the kernel start on boot?
>   A: Only one -- init
> 
> Apparently this isn't necessarily true any more, or soon won't be?

It's still true to a decent engineering approximation, assuming that you
don't count the pile of kernel threads that show up in the process
table, things like the kernel calling modprobe on demand sometimes, and
the initramfs probably muddies the waters a bit too. For the purposes of
a block-diagram-type view of the system this is as sensible an
assumption as it ever was, though, and Upstart certainly doesn't change
it.

> So, this brings us to upstart, the init replacement.  After a couple of 
> days of looking through both the on-line documentation I could find 
> (http://upstart.ubuntu.com/getting-started.html) and the actual 
> installed files  I find that I still have more than a number of 
> questions about how this is supposed to work currently and in the 
> future, as alluded to in this snippet from something posted to one of 
> the ubuntu devel lists:
> 
>     ------------
>     > Date: Fri, 26 Jun 2009 14:55:16 -0500
>     > From: Robbie Williamson <robbie at ubuntu.com>
>     > Subject: Debian/Ubuntu Boot Performance Sprint Summary
> 
>     <snip/>Scott James Remnant stated that he expects to have the
>     entire Ubuntu boot sequence transitioned over to upstart
>     by 10.04, with some work already planned for the 9.10 cycle.<snip/>
>     -------------
> 
> Questions:
> 1. the previously referenced site says that jobs files are placed in 
> /etc/init/jobs.d; actually, they seem to be in /etc/event.d -- what's up 
> with that?

Upstart 0.10, which is the series we plan to move to at least for 10.04
(though it's not clear whether that will happen for 9.10) will use
/etc/init/jobs.d/; the current version of Upstart in Ubuntu uses
/etc/event.d/. This went in tandem with a job file format change so it
isn't as gratuitous as it seems - the simultaneous move means that, this
time, we don't have to worry about mismatches between old/new Upstart
and new/old job files. (I think Scott mostly just preferred the new
location though.)

> 2.
> lizard:~~$ which init
> /sbin/init
> lizard:~~$ dpkg -S /sbin/init
> upstart: /sbin/init
> 
> So, I'm confused:  is init still being used, or what?

Upstart is an implementation of init. The kernel calls /sbin/init, so
that has to exist. It is not sysvinit, the older implementation of the
init daemon.

> /etc/inittab  no longer exists in 9.04/9.10,

This goes back as far as 6.10.

> Is this init an upstarted init?

Yes.

> 3. There's much discussion about improving the startup system by moving 
> to an event driven model.  The current implementation in 9.04/9.10alpha 
> simply mimics System-V init.  OK, this is transitional, but how are we 
> going to get from A to B?  Package implementers are not getting 
> initialization scripts right now (e.g. autofs), what's the plan for 
> rolling out a new event-driven system and how is it going to work?

Roughly, and note that I'm not responsible for implementing most of this
so errors and omissions excepted:

 * Move to a newer upstream version of Upstart to support new
   facilities, such as better daemon supervision and parsing LSB headers
   in init scripts
 * Move Upstart configuration to /etc/init/
 * Write debhelper command to install Upstart jobs and add a
   transitional upstart-job command to allow those jobs to be handled
   (at least to some extent) by the old sysv-rc system where that's
   necessary
 * Start moving core scripts over to Upstart jobs, and ensure that all
   remaining init scripts we ship include LSB headers
 * Once a reasonable number of core scripts have been moved over, switch
   Upstart over to dealing with the rc mechanism itself rather than via
   /etc/init.d/rc
 * Existing init scripts will still work; those that declare dependency
   information via LSB headers will be run once those dependencies are
   satisfied, while those that don't will be run at the end

The devil is in the detail, but this seems viable as a general plan. We
clearly can't do a flag day transition where everything moves over at
once, for all kinds of reasons.

> 4. Related: The concept of run levels is fairly useful, and the core 
> concept in System-V init.  Are run levels going to go way under full 
> upstart implementation?

Not to my knowledge. They remain useful, although since Upstart is more
general they don't need to be as deeply embedded into the implementation
there.

> And if run levels are not going away, how is process of startinɡ
> system daemons going to be event driven in any meaningful way?

Even in a fully event-driven system, run levels still usefully determine
what set of jobs you're *aiming* to start.

> 5. Even after reading through the "Getting Started" page twice, the 
> implementation under 9.04 is still confusing.  For example, to quote 
> Getting Started: "You list the events you want to start your job with 
> start on, and the events that stop your job with stop on."  OK, fair 
> enough.  However, here is /etc/event.d/rcS:
> 
>     ----------------
>     start on startup
> 
>     stop on runlevel
> 
>     console output
>     script
>        runlevel --set S >/dev/null || true
> 
>        PREVLEVEL=N
>        RUNLEVEL=S
>        export PREVLEVEL RUNLEVEL
> 
>        exec /etc/init.d/rcS
>     end script
>     ----------------
> 
> Doesn't this mean that rcS is stopped before /etc/init.d/rcS is ever 
> exec'd, since the runlevel is set before the script is run?  In 
> particular, when exactly does the runlevel event stop the script? 

The runlevel command just pokes the current runlevel into utmp so that
other things can fish it out; it doesn't emit the runlevel *event*.

What actually happens is that the rc-default job is declared as 'start
on stopped rcS', which runs when the script above (or, in practice,
/etc/init.d/rcS which it execs) exits; rc-default's script runs telinit;
telinit emits the runlevel event.

I think the 'stop on runlevel' is actually just there to make sure rcS
doesn't keep on trying to start, but Scott could clarify that.

> Again, package implementors can't get the timing down right now under
> System-V init.  This is going to turn into complete pandemonium if the 
> termination of events is ill-defined.

Your choice of words is interesting: "timing". To be fair, sysvinit made
it extremely hard to do anything on any basis other than time! Since
packagers are having to work around this lack by inserting things into
the correct order in numerical sequences, adding semi-random sleep
statements, and so forth it's honestly hardly surprising that they fail
to get it right. The problem that Upstart is trying to solve is
principally to make it possible and straightforward to robustly say that
some actions need other events to have happened: causality, not time. As
a packager who's had to deal with ordering concerns in the past (some of
which were circular in the System V design and we could only resolve
them by redesigning scripts to eliminate some constraints), I can hardly
wait.

I'm satisfied that the termination of events and other similar
state-diagram elements in Upstart are precisely defined, but they aren't
always well-*documented*. This will certainly have to be fixed before
Upstart is rolled out to the developer masses as an rc management system
as well as an init daemon.

> 6. "If you're using the example jobs, you will also have runlevel X 
> events, where X is one of 0–6 or S. Jobs will be run alongside the init 
> scripts for that runlevel."
> Presumably this means in an event driven model, user-installed jobs are 
> triggered in parallel with runlevel events?  Otherwise:  Um, what?

Right now it's tricky to declare a job that runs only in certain
runlevels, at least if it wants to have other dependencies too. As of
Upstart 0.5, there's an "and" operator for events which makes it
practical to say something like "start on runlevel 2 and started dbus".

> 7. Finally, is there better documentation available, say something in 
> between http://upstart.ubuntu.com/getting-started.html and slogging 
> through the actual source code?

Scott will have to answer this properly, I think. I can only manage an
oblique answer. :-)

To borrow some terminology, it's useful to divide the init system up
into three pieces: the init daemon, the rc mechanism, and the init
scripts. Right now, when looking at the version of Upstart in Ubuntu,
you're looking at a mature replacement init daemon and more or less a
prototype of the rc mechanism. It's got far enough to be able to express
basic system startup, but full conversion of init scripts really
requires something newer.

At the moment, we're not advising deployers to make use of the prototype
facilities for general user jobs precisely because we know we're going
to be changing them around in order to get to the point where they can
handle everything we need; we can cope with this when the job files
we're shipping are basically all quite tightly integrated with Upstart
itself, but it would make it pretty painful if we had to cope with
migrating lots of packages and even non-packaged jobs as well. Lack of
good daemon supervision in the version we're currently shipping also
means that you don't get as much benefit out of switching to Upstart
jobs as we're eventually expecting anyway. Deployers would be best
advised to hobble along with sysv-rc for a little while longer.

(For the avoidance of doubt, this isn't me arguing that the job format
shouldn't be better documented, but rather that if you're searching for
documentation because you want to write Upstart jobs and use them in
production rather than because you want to help with development, you
may be on the wrong track just now.)

Regards,

-- 
Colin Watson                                       [cjwatson at ubuntu.com]