Feature request: Graphical trace feature (patch included)

[Martin Lund]

Hi,

Attached is a patch which add a simple but powerful graphical trace 
feature to Upstart. It makes it possible to better optimize your Upstart 
init scripts.

The patch basically inserts a few trace functions into Upstart which 
collect time stamp information of job state changes and emitted events. 
This information is then arranged into a trace chart and dumped into a 
SVG file.

The attached SVG file is an example of such a trace dump. It shows a 
trace dump of an Ubuntu 10.10 desktop system starting. It can be viewed 
with just about any SVG viewer such as eog, display, inkscape etc..

Quick SVG explanation:
  * X-axis: Time (monotonic)
  * Y-axis: Job name
  * Blue lines represents user emitted events.
  * Red bars represents states: PRE-STOP, STOPPING, KILLED, POST-STOP.
  * Green bars represents states: STARTING, PRE-START, SPAWNED, POST-START
  * States RUNNING and WAITING are not depicted by color.

To enable the trace feature start the Upstart init process with the 
"--trace" parameter. After the system is finished starting simply fire 
the command "initctl trace-dump" which will tell the tracer to stop 
tracing and dump the SVG to "/tmp/upstart-trace.svg".

Added init trace options:
  --trace                            - enables trace functionality
  --trace-job-state-buffer  - set size of trace job state buffer 
(default = 2000)
  --trace-event-buffer       - set size of trace event buffer (default = 
2000)

Added initctl trace commands:
  trace-stop     - starts trace
  trace-start     - stops trace
  trace-dump   - dumps trace to /tmp/upstart-trace.svg (automatically 
stops trace)
  trace-reset    - resets trace data


Why SVG?

Answer: Its easy to generate. There is no loss of trace information as 
everything is represented by scalable vector elements. It allows viewers 
to zoom heavily into areas of interest. There are many SVG viewers 
available today and its easy to convert into beautiful bitmaps of any size.


Why not use bootchart etc. instead?

Answer: Don't stop using bootchart - this is not an alternative. 
Consider it a supplementary feature since the Upstart trace feature 
represents a different set of trace information. Bootchart and similar 
utilities does not relay Upstart specific information such as job state 
changes and emitted user events - this is exactly what this trace 
feature offers. Additionally, more Upstart specific information can be 
added in the future. It makes it possible to better optimize your 
Upstart init scripts. Also, its quite easy to enable and use.


I consider the patch to be lightweight as it is only about 1100 lines of 
code. Core trace functionality is about 700 lines of code. The remaining 
400 lines of code is mostly related to the command interface and some 
mandatory DBus code.

The patch was originally developed in a more basic form for Bang & 
Olufsen. They have successfully used it and continue to use it to 
optimize their Upstart init scripts in embedded systems. Somce then I 
have cleaned up and improved the patch a bit. It is made for Upstart 
0.6.5 but it should apply fairly easy to newer versions as well.

Please, try it out. Feel free to make any improvements or changes 
necessary.

Consider this a feature request to have the trace feature included in 
official Upstart - our favorite init system :)

Martin Lund

-- 
Martin Lund, Senior Software Consultant
DoréDevelopment ApS, Ved Stranden 1, 9560 Hadsund, DK-Denmark
Phone: +45 61 66 83 04, Email: mgl at doredevelopment.dk
WWW: http://www.doredevelopment.dk

-------------- next part --------------
A non-text attachment was scrubbed...
Name: upstart-0.6.5-trace-feature.patch.tar.gz
Type: application/gzip
Size: 8083 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/upstart-devel/attachments/20110124/f28a9430/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: upstart-trace-ubuntu.svg.tar.gz
Type: application/gzip
Size: 12696 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/upstart-devel/attachments/20110124/f28a9430/attachment-0003.bin>