Feature request: Graphical trace feature (patch included)

Tue Jan 25 10:09:51 UTC 2011

Hi Scott,

I'm glad you like the idea :)

I think the idea of exposing some of Upstarts internal states/mechanisms 
in a graphical trace chart is a useful thing especially when trying to 
obtain a detailed overview of the dynamic behavior of a systems init 
scripts. The main goal is of course to make it easy to spot faulty or 
poorly optimized init scripts and make it easy to do required 
optimization with regard to the various job states and events of Upstart.

You know the internals of Upstart better than I so maybe you can think 
of a few new things to expose in the trace in the future that might be 
useful or maybe even a better way to organize the SVG.

I'm sure that improvements can be made but at least I think this patch 
is a good starting point. It kind of demonstrates what is possible with 
relatively little code in terms of generating useful Upstart graphics.

I'm sorry that the demo SVG from my Ubuntu system seems a little 
compressed - it is because the example system takes only 5 seconds to 
finish booting! It is because my test system is a Virtualbox Ubuntu 
installation so everything is heavily precached and therefore it starts 
insanely fast - blame my vast amount of memory and fast i7 cpu :) . 
Also, notice that Ubuntus upstart-udev-bridge is kind of a spammer of 
user events which makes them harder to separate - in a clean embedded 
system it looks much nicer :)

Martin

On 2011-01-25 05:05, Scott James Remnant wrote:
> Hey Martin,
>
> I haven't had a chance to review the code yet, but the idea of this to
> me sounds awesome and I can't wait to try it out.
>
> Scott
>
> On Mon, Jan 24, 2011 at 1:09 PM, Martin Lund<mgl at doredevelopment.dk>  wrote:
>> Hi,
>>
>> Attached is a patch which add a simple but powerful graphical trace feature
>> to Upstart. It makes it possible to better optimize your Upstart init
>> scripts.
>>
>> The patch basically inserts a few trace functions into Upstart which collect
>> time stamp information of job state changes and emitted events. This
>> information is then arranged into a trace chart and dumped into a SVG file.
>>
>> The attached SVG file is an example of such a trace dump. It shows a trace
>> dump of an Ubuntu 10.10 desktop system starting. It can be viewed with just
>> about any SVG viewer such as eog, display, inkscape etc..
>>
>> Quick SVG explanation:
>>   * X-axis: Time (monotonic)
>>   * Y-axis: Job name
>>   * Blue lines represents user emitted events.
>>   * Red bars represents states: PRE-STOP, STOPPING, KILLED, POST-STOP.
>>   * Green bars represents states: STARTING, PRE-START, SPAWNED, POST-START
>>   * States RUNNING and WAITING are not depicted by color.
>>
>> To enable the trace feature start the Upstart init process with the
>> "--trace" parameter. After the system is finished starting simply fire the
>> command "initctl trace-dump" which will tell the tracer to stop tracing and
>> dump the SVG to "/tmp/upstart-trace.svg".
>>
>> Added init trace options:
>>   --trace                            - enables trace functionality
>>   --trace-job-state-buffer  - set size of trace job state buffer (default =
>> 2000)
>>   --trace-event-buffer       - set size of trace event buffer (default =
>> 2000)
>>
>> Added initctl trace commands:
>>   trace-stop     - starts trace
>>   trace-start     - stops trace
>>   trace-dump   - dumps trace to /tmp/upstart-trace.svg (automatically stops
>> trace)
>>   trace-reset    - resets trace data
>>
>>
>> Why SVG?
>>
>> Answer: Its easy to generate. There is no loss of trace information as
>> everything is represented by scalable vector elements. It allows viewers to
>> zoom heavily into areas of interest. There are many SVG viewers available
>> today and its easy to convert into beautiful bitmaps of any size.
>>
>>
>> Why not use bootchart etc. instead?
>>
>> Answer: Don't stop using bootchart - this is not an alternative. Consider it
>> a supplementary feature since the Upstart trace feature represents a
>> different set of trace information. Bootchart and similar utilities does not
>> relay Upstart specific information such as job state changes and emitted
>> user events - this is exactly what this trace feature offers. Additionally,
>> more Upstart specific information can be added in the future. It makes it
>> possible to better optimize your Upstart init scripts. Also, its quite easy
>> to enable and use.
>>
>>
>> I consider the patch to be lightweight as it is only about 1100 lines of
>> code. Core trace functionality is about 700 lines of code. The remaining 400
>> lines of code is mostly related to the command interface and some mandatory
>> DBus code.
>>
>> The patch was originally developed in a more basic form for Bang&  Olufsen.
>> They have successfully used it and continue to use it to optimize their
>> Upstart init scripts in embedded systems. Somce then I have cleaned up and
>> improved the patch a bit. It is made for Upstart 0.6.5 but it should apply
>> fairly easy to newer versions as well.
>>
>> Please, try it out. Feel free to make any improvements or changes necessary.
>>
>> Consider this a feature request to have the trace feature included in
>> official Upstart - our favorite init system :)
>>
>> Martin Lund
>>
>> --
>> Martin Lund, Senior Software Consultant
>> DoréDevelopment ApS, Ved Stranden 1, 9560 Hadsund, DK-Denmark
>> Phone: +45 61 66 83 04, Email: mgl at doredevelopment.dk
>> WWW: http://www.doredevelopment.dk
>>
>>
>> --
>> upstart-devel mailing list
>> upstart-devel at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/upstart-devel
>>
>>