Integrating LTTng in Ubuntu

Tue May 3 14:44:55 UTC 2011

On Tue, 2011-05-03 at 00:09 -0400, Julien Desfossez wrote:
> On 11-05-02 04:55 PM, Leann Ogasawara wrote:
> > On Mon, 2011-05-02 at 14:05 -0400, Julien Desfossez wrote:
> >> Hello,
> >>
> >> LTTng, the Linux Trace Toolkit Next Generation, is a project that aims
> >> at producing a highly efficient full system tracing solution. It is
> >> composed of several components to allow tracing of the kernel, of
> >> userspace, trace viewing and analysis and trace streaming.
> >>
> >> Ubuntu is more and more popular as a server-grade distribution and
> >> having a comprehensive tracing infrastructure built into the kernel
> >> would allow system administrators to investigate problems as they appear
> >> without patching the kernel and rebooting the server.
> >>
> >> For example, with LTTng you know really quickly what is the I/O
> >> bandwidth (network or disk) associated with a particular process, and in
> >> this process which file is the culprit for slowing down your whole
> >> system. LTTng can also be used as a highly efficient userspace tracer
> >> with UST. When the user-space and kernel-space traces are combined,
> >> application developpers and sysadmins have a very detailed overview of
> >> what is happening on their system.
> >>
> >> Over the last months, the size of the LTTng kernel patchset has been
> >> considerably shrinked for Linux distributions by removing all
> >> LTTng-specific instrumentation from the tree.
> >>
> >> You can find the latest patchset here [1].
> >>
> >> Please note that the patches within this patchset are elected as the
> >> first items from the LTTng project to eventually get into the mainline
> >> Linux kernel. This includes TRACE_EVENT() changes, the LTTng trace
> >> clocks and the Generic Ring Buffer Library. Please take into account
> >> that most of those patches are really small patches that have been split
> >> to facilitate mainlining by addressing them to the appropriate maintainers.
> >>
> >> Also, you will notice that most of the bigger patches are actually new
> >> functionalities that won't conflict with existing code base, because
> >> they create files in new directories of the kernel tree.
> >>
> >> The upcoming UDS would be a good timing to discuss the integration of
> >> the new LTTng as an efficient tracer especially for the server flavour
> >> of the Ubuntu kernel.
> >>
> >> I submitted a blueprint [2] and following the discussion with Leann
> >> Ogasawara I'm asking the Ubuntu Kernel-team their feedbacks and opinions
> >> about making a full fledged blueprint/spec. I will be in Budapest next week.
> > 
> > One of my concerns is the maintenance burden this brings upon the Ubuntu
> > Kernel Team.  The LTTng patch set looks like it's 86 patches at the
> > moment.  In comparison, our entire Ubuntu patch delta from the upstream
> > kernel when the Oneiric kernel opened was ~150 patches.
> Yes but if you look closely you will see that most of the patches are
> either small or add a new feature that doesn't conflict with the vanilla
> kernel.
> The 32 patches in the trace-event-semicolon-removal are split per file
> they modify and usually modify very few lines. Moreover these patches
> are being mainlined as we speak (really short ETA).

Does "really short ETA" imply the patches will land in the 2.6.40
kernel?  Are they already in a subsystem maintainer's tree?

> The lib-ring-buffer patches will enter in the mainline kernel as soon as
> we make perf work with it (around september).

How long do you think this work will take?  Which mainline kernel would
you anticipate them to land in?

> The trace-clock patches are architecture specific and work to mainline
> these is in progress also (all the ARM related patches are already in
> the Ubuntu kernel IIRC).

Same question, which upstream kernel version do you expect the
trace-clock patches to land?  2.6.40?  Also, I don't see any ARM related
trace-clock patches in the Ubuntu distro. 

> >> The Linaro project already ships with LTTng. For the next Ubuntu
> >> release, it would be really interesting to have a LTTng-ready kernel in
> >> archive to see if it could become the default kernel in the next LTS.
> >> Our mainlining expectations for the biggest patches are around september
> >> 2011.

So the reason I keep inquiring about which upstream kernel you
anticipate patches to land is because it's a good possibility Oneiric's
kernel will at least be a 2.6.40 based kernel.  If a good majority of
the patches land in 2.6.40 we can then re-evaluate what's left that
needs applied, if any.  In my opinion, I think you effort is best spent
getting the patches applied upstream this cycle.

> > If the plan is to get these into mainline around the September 2011 time
> > frame, why not just re-target this spec/blueprint to the Oneiric+1 dev
> > cycle?  We'd then have the patches incorporated with no additional
> > effort as they'd filter in via upstream.  Is there a compelling reason
> > for us to have to carry these patches prior to them landing upstream?
> Our hope is to have a complete working stack in Oneiric so that when the
> next LTS comes up, it will be completely ready. As I said, our target is
> mainly sysadmins, they will enjoy having a server LTS release with all
> the tracing infrastructure they need fully tested, documented and ready,
> that's why it could be interesting to have it enabled in the release
> pre-LTS while we work on mainlining the remaining patches.
> It would be great to have Oneiric with a LTTng kernel by default, but if
> you think it's too much work, what about having an alternate source
> kernel (like Linaro's) in Universe or have some kind of official PPA for
> it ?

You are more than welcome to produce an LTTng kernel in Universe or
available for testing in a PPA but I can't commit any Ubuntu kernel team
resources for that.

Thanks,
Leann