A Proposal to deal with groups

[Saravanan Shanmugham (sarvi)]

Point taken about use cases.

The concept of complex states(with out processes) is certainly a very
usefull one going forward and has its own use cases such as the ones you
refer.  
But I don't believe complex states is the same as groups.

I consider groups very different from state, though it makes sense that
a group also have state that is some combination of the individual jobs
in the group. 
Groups in my mind are a convenient way of doing an operation on a
collection of jobs without knowing the individual jobs names.

What I am running into are usecases where
   1. what constitues a group ends up being dynamic. When a package gets
added/deleted there are jobs in that package that need to join or leave
the group, without having to modify some central file. So in that sense,
though we can define a group as a job file where we can list all the
elements of the group as 
       while job1 and job2 and job3
      This model doesn't allow that
   2. Then the question of how should the individual states of jobs in a
group affect the group state. This is usualy either AND or OR but never
a mix of them within a general group definition as it doesn't make sense
for each member job to join the group with a different effect on the
group state.
   3. Then comes the question of what does it mean to do an operation on
a group. Stopping a group can be defined as stopping all jobs in the
group some of which may be running and others not. But then what does it
mean to start the group. 
   4. Last is the concept of multiple "virual process managers". This is
a different sort of Group that is not addressed in my powerpoint. Where
event interactions are between jobs in this "virtual process manager"
group and does not affect jobs outside the virtual process manager
group. This has its usecases in virtualization or partitioning a process
and insulating them from others.  

Here are some of the scenarios we want to handle.
    Use case 1: We need to group jobs to be able to operate on them as a
set.   
          For us there are a group of processes that may need to be
restarted when one or more "critical processes" crash/restart. What
constitutes a "critical processes" as well as what contitutes the group
that needs restarting, may change over time as one or more packages in
the system are upgraded/downgraded or added/deleted.  

     Use case 2: a package may have 10 jobs as part of the package.
Before I remove the package I need to be able to stop all jobs
associated with the package. Yes we can try to keep a list of all jobs
in the package. But this is also a case where concept of group could
help me keep it simple.

     Use Case 3: Virtual process managers or partiitions

Thanks,

Sarvi

-----Original Message-----
From: Scott James Remnant [mailto:scott at netsplit.com] 
Sent: Thursday, July 23, 2009 3:29 AM
To: Saravanan Shanmugham (sarvi)
Cc: Upstart Development
Subject: Re: A Proposal to deal with groups
Importance: High

On Wed, 2009-07-22 at 23:36 -0700, Saravanan Shanmugham (sarvi) wrote:

> I have been struggling with what it means to have groups in Upstart 
> and what the expectd behaviour would be.
>  
> Here is what I have come up. I went through several iterations over 
> the last few months thinking about what it should look like.
> This is the one that feels good so far or anywhere close to it.
>  
> Here is a proposal to support groups in Upstart.
> 
Hi Sarvi,

Thanks for your work on this!

One thing I've found that has helped figuring out features and designs
for Upstart is to set down the use cases first.  Your presentation
focuses on the details of how you'd implement groups, without putting
them into a context of how they would be used.

It'd certainly help people reading it (including me!) to understand
where you're coming from.

Another useful reason for having use cases is when you have two possible
implementations, and want to find out which works best.


I'm not quite sure whether Groups are the same thing as States or a
separate first-class object.  It's probably a good idea for me to write
down the direction I've been vaguely heading, so we can figure out
whether I've got this wrong ;-)

The use cases for states are:

 * define system states ("networking"), ("multi-user") based on the
   availability of a set of services or completion of a set of tasks

 * define more advanced states based on things like hardware
   ("on ac power"), ("docked") and have services included or excluded
   from these states

 * define groups of jobs together as a single set ("LAMP stack") where
   the set is automatically marked active when its components are active

   thus a sysadmin can use "status lamp" to see whether the LAMP stack
   is active, rather than checking the individual pieces

 * combine with "while" to allow a sysadmin to start a state


The implementation is simply to have process-less jobs.  This has been
permitted in Upstart since 0.3, a process-less job remains in the
running state until it is stopped.

Thus a state is simply a file containing a "while" clause, multi-user
could be (in a sysv-compatible system):

   while runlevel RUNLEVEL=[2345]

Things like ac power, docked, would use different events/states as a
base.

LAMP might look like:

   while apache and mysql


When apache and mysql are started, the LAMP state would be started too
(while condition met) so show as running; if either apache or mysql were
stopped, the LAMP state would be stopped too (while condition no longer
met) and show as stopped.

Other services can then use "while lamp", etc.


The "start" and "stop" sysadmin commands have a special behaviour, if
the while condition is not true, but the members that are false are
Upstart jobs, it will pre-start those.  So a sysadmin may use "start
lamp"


This merging is why 0.6 uses a single directory of *.conf files, each
file specifies a task, service or state and the difference is just the
content.

If I've got this wrong, we'll probably want multiple directories again
or multiple extensions.

Scott
--
Have you ever, ever felt like this?
Had strange things happen?  Are you going round the twist?