[CoLoCo] Job/batch management

[David J W]

Hello,
    First off an introduction as this is my first post:

      I work in Denver as a multi-hat developer: php/python/bash/c++
developer, linux system admin, and a intermediate MySQL database
administrator, and also my current company's R&D developer.  I've been
working with *nix since 1996-97 when a c++ programming course I took was
taught exclusively on v220 terminals connected to a mini-computer.

I'm on week 2 or maybe 3 of doing research for consolidating my company's
small and growing horde of cronjobs... its well over 150 script at this
point.  The company CEO has directed me to start research and research I
have done.

Why crontab sucks for what I need:

  Cron does a fantastic job at what it does, run something at X time.
  The problem is, what if I need alt_X to run if X exists non-zero....
alright so I could do:
    x || alt_x

   Now what if I need y to run after x if its successfull

   (x && y ) || alt_x

   Well thats not right?  x could perform correctly then have y fail,
causing alt_x to run and possibly redo important operations/data.

   (x || alt_x) && y  - except what if I don't want y to run if alt_x ends
up running?


   Now to make things fun.... lets say there is a chain of 4-5 scripts that
must run conditionally.  I'm pretty decent with simple and/or logic, but I
know this would confuse anyone else to read and decypher.  Making it so only
I and maybe one or two disciples can maintain the abomination that is the
company's backend underworld.

Plan B (partially in effect)
   Complex jobs are written as bash scripts.  This is a step up from the
above, I can build a job with variables, capture output to temp files, and
use mail or sendmail to make customized emails.  Whats lacking is
resuability.  Sure I can write shell script libraries to reuse certain
things, like a bash wrapper function to handle individual scripts.... but
adding in context sensitive checks "has the word failure, critical, OMFG,
etc" would be hard coded in.

Plan C ( what I am aiming for)
   So I'm looking for a cronjob equivalent scheduling system that can be
configured by information stored in a database (MySQL, sqllite, whatever).
This phantom program would have the following specs:

    Run any process/job as either root or a specific user account
    Record the job fitness ( maximum memory, time stats (rtime, utime,
overall)
    Historical recording ( save input to DB or serialized filestore )
    Context sensitive output handling
    Digest email capability ( instead of 120-140 emails a day, I'd be
estatic with like 30)
    SMP/pager capability

   I've started prototyping the above in python, but I figure to get this to
a stable enough product, it would take me 3-4 days of development time then
a residual 5-15 hours for the next month or two in responsive debugging.