Integrating LTTng in Ubuntu

[Julien Desfossez]

On 11-05-02 04:55 PM, Leann Ogasawara wrote:
> On Mon, 2011-05-02 at 14:05 -0400, Julien Desfossez wrote:
>> Hello,
>>
>> LTTng, the Linux Trace Toolkit Next Generation, is a project that aims
>> at producing a highly efficient full system tracing solution. It is
>> composed of several components to allow tracing of the kernel, of
>> userspace, trace viewing and analysis and trace streaming.
>>
>> Ubuntu is more and more popular as a server-grade distribution and
>> having a comprehensive tracing infrastructure built into the kernel
>> would allow system administrators to investigate problems as they appear
>> without patching the kernel and rebooting the server.
>>
>> For example, with LTTng you know really quickly what is the I/O
>> bandwidth (network or disk) associated with a particular process, and in
>> this process which file is the culprit for slowing down your whole
>> system. LTTng can also be used as a highly efficient userspace tracer
>> with UST. When the user-space and kernel-space traces are combined,
>> application developpers and sysadmins have a very detailed overview of
>> what is happening on their system.
>>
>> Over the last months, the size of the LTTng kernel patchset has been
>> considerably shrinked for Linux distributions by removing all
>> LTTng-specific instrumentation from the tree.
>>
>> You can find the latest patchset here [1].
>>
>> Please note that the patches within this patchset are elected as the
>> first items from the LTTng project to eventually get into the mainline
>> Linux kernel. This includes TRACE_EVENT() changes, the LTTng trace
>> clocks and the Generic Ring Buffer Library. Please take into account
>> that most of those patches are really small patches that have been split
>> to facilitate mainlining by addressing them to the appropriate maintainers.
>>
>> Also, you will notice that most of the bigger patches are actually new
>> functionalities that won't conflict with existing code base, because
>> they create files in new directories of the kernel tree.
>>
>> The upcoming UDS would be a good timing to discuss the integration of
>> the new LTTng as an efficient tracer especially for the server flavour
>> of the Ubuntu kernel.
>>
>> I submitted a blueprint [2] and following the discussion with Leann
>> Ogasawara I'm asking the Ubuntu Kernel-team their feedbacks and opinions
>> about making a full fledged blueprint/spec. I will be in Budapest next week.
> 
> One of my concerns is the maintenance burden this brings upon the Ubuntu
> Kernel Team.  The LTTng patch set looks like it's 86 patches at the
> moment.  In comparison, our entire Ubuntu patch delta from the upstream
> kernel when the Oneiric kernel opened was ~150 patches.
Yes but if you look closely you will see that most of the patches are
either small or add a new feature that doesn't conflict with the vanilla
kernel.
The 32 patches in the trace-event-semicolon-removal are split per file
they modify and usually modify very few lines. Moreover these patches
are being mainlined as we speak (really short ETA).
The lib-ring-buffer patches will enter in the mainline kernel as soon as
we make perf work with it (around september).
The trace-clock patches are architecture specific and work to mainline
these is in progress also (all the ARM related patches are already in
the Ubuntu kernel IIRC).

>> The Linaro project already ships with LTTng. For the next Ubuntu
>> release, it would be really interesting to have a LTTng-ready kernel in
>> archive to see if it could become the default kernel in the next LTS.
>> Our mainlining expectations for the biggest patches are around september
>> 2011.
> 
> If the plan is to get these into mainline around the September 2011 time
> frame, why not just re-target this spec/blueprint to the Oneiric+1 dev
> cycle?  We'd then have the patches incorporated with no additional
> effort as they'd filter in via upstream.  Is there a compelling reason
> for us to have to carry these patches prior to them landing upstream?
Our hope is to have a complete working stack in Oneiric so that when the
next LTS comes up, it will be completely ready. As I said, our target is
mainly sysadmins, they will enjoy having a server LTS release with all
the tracing infrastructure they need fully tested, documented and ready,
that's why it could be interesting to have it enabled in the release
pre-LTS while we work on mainlining the remaining patches.
It would be great to have Oneiric with a LTTng kernel by default, but if
you think it's too much work, what about having an alternate source
kernel (like Linaro's) in Universe or have some kind of official PPA for
it ?

Thanks,

Julien