That upstart Upstart

[Patrick Goetz]

(Apologies in advance for the length of this message -- if you don't 
care about order and manner in which system daemons are launched, please 
invoke the Delete key operator immediately.)

Everyone knows the canonical (no pun intended) unix interview question, 
namely

  Q: how many processes does the kernel start on boot?
  A: Only one -- init

Apparently this isn't necessarily true any more, or soon won't be?  I 
usually don't dig into random subsystems until something fails.  In this 
case, I recently found out that my coworker had "fixed" an autofs 
startup timing problem in our 8.04 rollout by simply inserting a few 
sleep loops into the autofs init script.  This hack was no longer 
working in 9.04, which we're testing now (not to mention dramatically 
increased boot times).  Since autofs is a core system service, I 
suggested that he contact Canonical support and let them know that there 
is a problem; i.e. as installed by default, autofs fails to start.  The 
support technician who got back to us made a number of comments about 
how processes are launched which were simply wrong.  At this point, 
Houston: we have a problem.  The Ubuntu tech staff isn't even clear on 
how daemons are launched at startup.

So, this brings us to upstart, the init replacement.  After a couple of 
days of looking through both the on-line documentation I could find 
(http://upstart.ubuntu.com/getting-started.html) and the actual 
installed files  I find that I still have more than a number of 
questions about how this is supposed to work currently and in the 
future, as alluded to in this snippet from something posted to one of 
the ubuntu devel lists:

    ------------
    > Date: Fri, 26 Jun 2009 14:55:16 -0500
    > From: Robbie Williamson <robbie at ubuntu.com>
    > Subject: Debian/Ubuntu Boot Performance Sprint Summary

    <snip/>Scott James Remnant stated that he expects to have the
    entire Ubuntu boot sequence transitioned over to upstart
    by 10.04, with some work already planned for the 9.10 cycle.<snip/>
    -------------

Questions:
1. the previously referenced site says that jobs files are placed in 
/etc/init/jobs.d; actually, they seem to be in /etc/event.d -- what's up 
with that?

2.
lizard:~~$ which init
/sbin/init
lizard:~~$ dpkg -S /sbin/init
upstart: /sbin/init

So, I'm confused:  is init still being used, or what?  /etc/inittab  no 
longer exists in 9.04/9.10, and the jobs in /etc/event.d follow the 
event driven model outlined in the documentation, however there is still 
an init executable.  Is this init an upstarted init?  There don't appear 
to be any other candidate executables in the upstart package.

3. There's much discussion about improving the startup system by moving 
to an event driven model.  The current implementation in 9.04/9.10alpha 
simply mimics System-V init.  OK, this is transitional, but how are we 
going to get from A to B?  Package implementers are not getting 
initialization scripts right now (e.g. autofs), what's the plan for 
rolling out a new event-driven system and how is it going to work?

4. Related: The concept of run levels is fairly useful, and the core 
concept in System-V init.  Are run levels going to go way under full 
upstart implementation?  If so, how will the system distinguish between, 
say, a maintenance single-user boot and what is currently rc2?  And if 
run levels are not going away, how is process of startinɡ system daemons 
going to be event driven in any meaningful way?

5. Even after reading through the "Getting Started" page twice, the 
implementation under 9.04 is still confusing.  For example, to quote 
Getting Started: "You list the events you want to start your job with 
start on, and the events that stop your job with stop on."  OK, fair 
enough.  However, here is /etc/event.d/rcS:

    ----------------
    start on startup

    stop on runlevel

    console output
    script
       runlevel --set S >/dev/null || true

       PREVLEVEL=N
       RUNLEVEL=S
       export PREVLEVEL RUNLEVEL

       exec /etc/init.d/rcS
    end script
    ----------------

Doesn't this mean that rcS is stopped before /etc/init.d/rcS is ever 
exec'd, since the runlevel is set before the script is run?  In 
particular, when exactly does the runlevel event stop the script? 
Clearly after the rcS script is exec'd (otherwise none of the goodies in 
/etc/rc2.d would ever get started), but this needs to be documented much 
more carefully, since not every case will be this clear cut.  Again, 
package implementors can't get the timing down right now under System-V 
init.  This is going to turn into complete pandemonium if the 
termination of events is ill-defined.

6. "If you're using the example jobs, you will also have runlevel X 
events, where X is one of 0–6 or S. Jobs will be run alongside the init 
scripts for that runlevel."
Presumably this means in an event driven model, user-installed jobs are 
triggered in parallel with runlevel events?  Otherwise:  Um, what?

7. Finally, is there better documentation available, say something in 
between http://upstart.ubuntu.com/getting-started.html and slogging 
through the actual source code?