[PATCH] Add 'log' argument to 'console' stanza allowing job output to be captured.

[James Hunt]

Hi Jacek,

On 28/10/11 08:53, Jacek Konieczny wrote:
> On Thu, Oct 27, 2011 at 05:36:49PM +0100, James Hunt wrote:
>> This patch adds a 'log' argument to the 'console' stanza which allows all job output to be captured
>> and persisted to disk.
> 
> Are there any limits on the log sizes?
Not currently (the logger is at this stage very simple).

> Any log rotation procedure?
There are already tools such as logrotate(8) that handle log rotation. By default Upstart will write
system logs to /var/log/upstart/ so logrotate can be configured to deal with logs in that directory
appropriately.

> A way
> to forward the stream to syslog?
Again, not currently, but feel free to raise a bug so we could consider that as an enhancement.

> 
>> Note that this patch makes "console log" the new default (the old default being "console none"). The
>> rationale for this change being that if your jobs do produce output, you generally want to see it
>> (more than likely it will contain details of errors). If you do not want this behaviour, you will
>> need to modify your job configuration files to specify explicitly "console none". This is a change
>> in behaviour. This feature is available to user jobs as well [*].
> 
> In case a job produces some insane amount of output (which could be
> unnoticed with older Upstart) won't this fill up the disk, possibly
> causing deny of service?
There are 3 issues here: disk, memory and badly behaved jobs:

1) Disk

Yes it would be possible for a job to fill a disk (partition). But that's true of any application
that can write to disk and has sufficient quota. Again, that's where tools like logrotate help, and
having servers with separate disk partitions to avoid a major impact on other system aspects.


2) Memory

We should improve the failure case: ENOSPC should be special-cased such that if Upstart detects
there is insufficient space, it does not attempt to store any more "unflushed data" internally. It
should probably free the oldest unflushed data and store the new data such that if space
subsequently becomes available it will atleast write the _latest_ data to disk (in other words a
ring buffer). Alternatively, we could have it:

- display a warning for the job
- discard all unflushed job data
- revert the job to "console none"

This would be safer but potentially annoying for users who allowed their disks to fill as they'd
never see the job output.


3) Badly Behaved Jobs

If a job or a daemon is producing heaps of data, that needs to be investigated. The logger will
expose these badly behaved jobs to allow them to be fixed :)

We could change the console default back to "none", but if you were testing this feature (which of
course you should before using in anger), I'd argue its less effort to add "console none" to one or
two of your jobs that were behaving badly than to modify every job to say "console log". Since most
daemons don't produce any output and if they do you generally want to be aware of it, "console log"
seems a reasonable default. One of my test machines has 99 jobs. Of those, only 16 are producing
output and only 1 is what we'd term "badly behaved" (and I'll be raising a bug on that one today).

For symmetry with other recent changes, we should probably have a '--no-log' option which would
disable all job logging (system jobs + user jobs) by:

- reverting all jobs to have the old console default of "console none".
- permit "console log" but treat it as "console none".

For full flexibility we could also add a '--default-console <value>' option maybe to allow
'--default-console none'? This would be different to '--no-log' in that if a job does specify
"console log" explicitly, only that's jobs output would be logged. If that sounds useful, please
raise a bug.

By the way, the intention is for a follow-on patch to add the ability to specify something like:

  console log output

Such that all job output would be logged *and* sent to the console. This would be useful for jobs
like mountall and the SysV jobs. That is the only combination that makes any sense (in particular
"console log owner" could leak sensitive information to the log).

Thanks for your comments. It is worth stating that this feature allows Upstart to write files, which
it has *never* done before. So, we do need to be consider all the edge cases carefully.

Kind regards,

James.
--
James Hunt
____________________________________
http://upstart.ubuntu.com/cookbook
http://upstart.ubuntu.com/cookbook/upstart_cookbook.pdf