That upstart Upstart

Scott James Remnant scott at canonical.com
Tue Jun 30 10:42:19 BST 2009


On Mon, 2009-06-29 at 16:11 -0500, Patrick Goetz wrote:

> Everyone knows the canonical (no pun intended) unix interview question, 
> namely
> 
>   Q: how many processes does the kernel start on boot?
>   A: Only one -- init
> 
> Apparently this isn't necessarily true any more, or soon won't be?
> 
In the literal sense, it's still true.  Once the kernel initialisation
has completed, it runs the /sbin/init executable on the root filesystem.
It's up to this executable to run all of the other processes on the
system.

When using System V init, this /sbin/init binary is provided by the
sysvinit package.  When using Upstart, this binary is provided by the
upstart package.

While they have different configuration and operation, they are still
fundamentally a single binary from which all other processes are
spawned.


Now, I said literal sense, because on a modern system that's not
entirely true.  This has nothing to do with Upstart, but simply the way
that Linux has evolved.

Firstly the root filesystem's /sbin/init is rarely the first process
started these days.  Instead the /init executable inside a boot-loaded
supplied "initial ram filesystem" (initramfs) is actually run; this is
then responsible for mounting to the root filesystem, and the final
thing it does is exec /sbin/init itself.

Thus the new /sbin/init on the root filesystem still has pid #1.


Secondly, the kernel isn't just a monolithic processes these days but
has many threads of operation.  These threads are effectively processes
being run inside kernel space, some are even userspace from all
effective points of view other than the fact they were spawned by the
kernel.

Many of these are spawned before init is actually started, but the
kernel takes care to assign them pids beginning 2.


Thirdly the kernel itself will spawn userspace processes directly
itself; the original example of this is that the kernel will
execute /sbin/modprobe itself if you open a device node that has no
attached driver (passing a magic alias along the lines of
char-major-123-4).

Other examples are the defunct /sbin/hotplug tool, and apport which is
run when a process core dumps.

Since these are executed out the kernel, they fall outside of the
"everything is run by init" model.

> So, this brings us to upstart, the init replacement.  After a couple of 
> days of looking through both the on-line documentation I could find 
> (http://upstart.ubuntu.com/getting-started.html) and the actual 
> installed files  I find that I still have more than a number of 
> questions about how this is supposed to work currently and in the 
> future, as alluded to in this snippet from something posted to one of 
> the ubuntu devel lists:
> 
>     ------------
>     > Date: Fri, 26 Jun 2009 14:55:16 -0500
>     > From: Robbie Williamson <robbie at ubuntu.com>
>     > Subject: Debian/Ubuntu Boot Performance Sprint Summary
> 
>     <snip/>Scott James Remnant stated that he expects to have the
>     entire Ubuntu boot sequence transitioned over to upstart
>     by 10.04, with some work already planned for the 9.10 cycle.<snip/>
>     -------------
> 
Great!  Always happy to answer any questions about it! :-)

> Questions:
> 1. the previously referenced site says that jobs files are placed in 
> /etc/init/jobs.d; actually, they seem to be in /etc/event.d -- what's up 
> with that?
> 
Upstart is in heavy development, and things are subject to change until
1.0 is out.  Once that happens, it'll all be stable and compatible, but
up to that point I would rather get things right than be bound to not
change things.

The document you're reading is from the upstream site, and thus actually
documents a later version of Upstart (0.5) than is currently in Ubuntu
(0.3)

I should probably point out that the next version (0.6) uses a different
location again.  These location changes generally come with a job format
change, so it does actually make migration easier.

Why the change at all?  /etc/event.d was causing a lot of confusion,
people started to think that jobs were events - when the two are quite
different.  The original reason for using /etc/event.d was that it
fitted the /etc/dbus-1/event.d naming for "things run on events".

Future versions will just use /etc/init

> 2.
> lizard:~~$ which init
> /sbin/init
> lizard:~~$ dpkg -S /sbin/init
> upstart: /sbin/init
> 
> So, I'm confused:  is init still being used, or what? 
> 
Yes.

Upstart is an implementation of /sbin/init, thus the binary it installs
is still called /sbin/init (since that's what the kernel runs).

--version reveals all here:

----8<--------8<--------8<--------8<--------8<--------8<--------8<----
littlebigplanet scott% /sbin/init --version
init (upstart 0.6.0)
Copyright (C) 2009 Canonical Ltd.

This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
---->8-------->8-------->8-------->8-------->8-------->8-------->8----

> /etc/inittab  no longer exists in 9.04/9.10, and the jobs in /etc/event.d
> follow the event driven model outlined in the documentation, however there
> is still an init executable.  Is this init an upstarted init?  There don't
> appear to be any other candidate executables in the upstart package.
> 
Correct.

All of the traditional init-related executables, including /sbin/init,
are supplied by the upstart package and are different implementations
than those supplied by the sysvinit packahe.

> 3. There's much discussion about improving the startup system by moving 
> to an event driven model.  The current implementation in 9.04/9.10alpha 
> simply mimics System-V init.  OK, this is transitional, but how are we 
> going to get from A to B?  Package implementers are not getting 
> initialization scripts right now (e.g. autofs), what's the plan for 
> rolling out a new event-driven system and how is it going to work?
> 
It's a multi-stage plan.

The first of which has been to get the init daemon right.  That's taken
a bit longer than planned because I got promoted to management, and
failed to find any time to continue working on it.  I'm fully recovered
now, and a developer once more, so am once again working full time.

Upstart's still not quite right, the 0.5 series was much more robust
than the 0.3 series but the dependency on D-Bus introduced userland
issues.  The 0.6.0 release fixes those, so will replace 0.3 and 0.5
wherever either are being used.

But 0.6 still has underlying model issues that need to be addresses, the
0.10 release coming later this year (around September time) is intended
to correct those.

0.6 is a great time to start converting some jobs over, 0.10 is the best
time to convert them all.


As to the conversion, well firstly there's no rush.  Upstart supports a
superset of sysvinit's functionality, so is able to run the sysv-rc
script just as well as sysvinit could.

The sysv-rc script (/etc/init.d/rc) is the one that takes care of
following all those /etc/rc?.d/S??* and K??* symlinks and starting and
stopping their services.

This happens alongside Upstart's event-driven model.

Thus the easiest way to start is by converting those bottom-layer and
leaf-layer services.


The next step is to introduce a bit of harmony between the two.  The
plan to do this was recently agreed with Debian, and looks something
like this:

 - packages converted to contain Upstart jobs will ship the Upstart job
   as normal

 - in /etc/init.d they will ship a symlink pointing to
   the /lib/init/upstart-job "helper"

 - this helper will run "start <job>"

Thus there's a nice transition period where people can still use the old
init script path.

Handily, the sysv-rc script (or insserv/startpar) will know about these
symlinks and will run them as if they were init scripts.  This will
issue a start command to Upstart.

If the job's already running, that's a no-op.

If the job's not running, it will start it.

This provides a nice, simple, way of allowing LSB init scripts to
"depend" on Upstart jobs.


The third step is to remove the sysv-rc script from the process.
Upstart will parse the init scripts in /etc/init.d itself and read the
LSB comment headers from them.

Based on these, it will synthesise jobs and their dependencies will
work.  This allows free inter-dependency between Upstart jobs and init
scripts, in both directions.

Since the 0.10 branch of Upstart will be able to supervise daemons, and
other processes that fork(), it will be able to track the pid of the
services started by the init script.

Thus even the Upstart "status" command will work.

> 4. Related: The concept of run levels is fairly useful, and the core 
> concept in System-V init.  Are run levels going to go way under full 
> upstart implementation?  If so, how will the system distinguish between, 
> say, a maintenance single-user boot and what is currently rc2?  And if 
> run levels are not going away, how is process of startinɡ system daemons 
> going to be event driven in any meaningful way?
> 
They're not going away, since they are required for LSB compatibility:

http://refspecs.freestandards.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/runlevels.html

Even if (as we hope) a future version of the LSB will mandate Upstart,
we'll still want to be able to provide backwards compatibility with
previous versions of the LSB.


That being said, runlevels only need apply to init scripts.  It's
perfectly fine for Upstart-native jobs to not worry about them.

This is because Upstart is far more flexible about defining system
states.  For example, I can define a "LAMP" state that is true when the
set of services are running (note 0.10-style config here):

    while apache and mysql

The nice thing about this is that the model works in both directions:

    # status apache
    apache stop/waiting
    # status mysql
    mysql stop/waiting
    # status lamp
    lamp stop/waiting

    # start apache
    apache start/running, process 1234
    # start mysql
    mysql start/running, process 1345

    # status lamp
    lamp start/running

And:

    # status apache
    apache stop/waiting
    # status mysql
    mysql stop/waiting
    # status lamp
    lamp stop/waiting

    # start lamp
    lamp start/running

    # status apache
    apache start/running, process 1234
    # status mysql
    mysql start/running, process 1345

Since you can define arbitrarily flexible system states, it's trivial to
provide "single user", "multiuser without network", "multiuser with
network" and "X server running" states.

The runlevel events and states can then simply be slaved off these, just
as they're slaved off each other today.

> 5. Even after reading through the "Getting Started" page twice, the 
> implementation under 9.04 is still confusing.  For example, to quote 
> Getting Started: "You list the events you want to start your job with 
> start on, and the events that stop your job with stop on."  OK, fair 
> enough.  However, here is /etc/event.d/rcS:
> 
>     ----------------
>     start on startup
> 
>     stop on runlevel
> 
>     console output
>     script
>        runlevel --set S >/dev/null || true
> 
>        PREVLEVEL=N
>        RUNLEVEL=S
>        export PREVLEVEL RUNLEVEL
> 
>        exec /etc/init.d/rcS
>     end script
>     ----------------
> 
> Doesn't this mean that rcS is stopped before /etc/init.d/rcS is ever 
> exec'd, since the runlevel is set before the script is run?
> 
No, the /sbin/runlevel tool merely looks up the current runlevel
in /var/run/utmp and when called with --set adds a new runlevel entry to
utmp.

The "telinit" tool is the one that actually emits the "runlevel" event.

If it helps any, this confusion is being fixed in 0.6, the telinit tool
takes care of all of the /var/run/utmp setting - as well as the need to
set the $PREVLEVEL and $RUNLEVEL environment variables.

> 6. "If you're using the example jobs, you will also have runlevel X 
> events, where X is one of 0–6 or S. Jobs will be run alongside the init 
> scripts for that runlevel."
> Presumably this means in an event driven model, user-installed jobs are 
> triggered in parallel with runlevel events?  Otherwise:  Um, what?
> 
They can be triggered wherever you want.

For example, if you want a job running before processing the multi-user
runlevels:

    start on starting rc[2345]

Your job will be running before the rc jobs are allowed to start.

Likewise:

    start on stopped rc[2345]

will run your job after the existing scripts have finished.

> 7. Finally, is there better documentation available, say something in 
> between http://upstart.ubuntu.com/getting-started.html and slogging 
> through the actual source code?
> 
Not yet, documentation is coming ;)  it's just harder to write than
code!


But please, if you have any questions, don't hesitate to ask.  Sometimes
being made to explain things helps me write documentation <g>

Scott
-- 
Scott James Remnant
scott at canonical.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/ubuntu-devel-discuss/attachments/20090630/499a0519/attachment-0001.pgp 


More information about the Ubuntu-devel-discuss mailing list