Integrating LTTng in Ubuntu

Tue May 3 21:11:03 UTC 2011

* Julien Desfossez (ju at klipix.org) wrote:
> On 11-05-03 10:44 AM, Leann Ogasawara wrote:
> > On Tue, 2011-05-03 at 00:09 -0400, Julien Desfossez wrote:
> >> On 11-05-02 04:55 PM, Leann Ogasawara wrote:
> >>> On Mon, 2011-05-02 at 14:05 -0400, Julien Desfossez wrote:
> >>>> Hello,
> >>>>
> >>>> LTTng, the Linux Trace Toolkit Next Generation, is a project that aims
> >>>> at producing a highly efficient full system tracing solution. It is
> >>>> composed of several components to allow tracing of the kernel, of
> >>>> userspace, trace viewing and analysis and trace streaming.
> >>>>
> >>>> Ubuntu is more and more popular as a server-grade distribution and
> >>>> having a comprehensive tracing infrastructure built into the kernel
> >>>> would allow system administrators to investigate problems as they appear
> >>>> without patching the kernel and rebooting the server.
> >>>>
> >>>> For example, with LTTng you know really quickly what is the I/O
> >>>> bandwidth (network or disk) associated with a particular process, and in
> >>>> this process which file is the culprit for slowing down your whole
> >>>> system. LTTng can also be used as a highly efficient userspace tracer
> >>>> with UST. When the user-space and kernel-space traces are combined,
> >>>> application developpers and sysadmins have a very detailed overview of
> >>>> what is happening on their system.
> >>>>
> >>>> Over the last months, the size of the LTTng kernel patchset has been
> >>>> considerably shrinked for Linux distributions by removing all
> >>>> LTTng-specific instrumentation from the tree.
> >>>>
> >>>> You can find the latest patchset here [1].
> >>>>
> >>>> Please note that the patches within this patchset are elected as the
> >>>> first items from the LTTng project to eventually get into the mainline
> >>>> Linux kernel. This includes TRACE_EVENT() changes, the LTTng trace
> >>>> clocks and the Generic Ring Buffer Library. Please take into account
> >>>> that most of those patches are really small patches that have been split
> >>>> to facilitate mainlining by addressing them to the appropriate maintainers.
> >>>>
> >>>> Also, you will notice that most of the bigger patches are actually new
> >>>> functionalities that won't conflict with existing code base, because
> >>>> they create files in new directories of the kernel tree.
> >>>>
> >>>> The upcoming UDS would be a good timing to discuss the integration of
> >>>> the new LTTng as an efficient tracer especially for the server flavour
> >>>> of the Ubuntu kernel.
> >>>>
> >>>> I submitted a blueprint [2] and following the discussion with Leann
> >>>> Ogasawara I'm asking the Ubuntu Kernel-team their feedbacks and opinions
> >>>> about making a full fledged blueprint/spec. I will be in Budapest next week.
> >>>
> >>> One of my concerns is the maintenance burden this brings upon the Ubuntu
> >>> Kernel Team.  The LTTng patch set looks like it's 86 patches at the
> >>> moment.  In comparison, our entire Ubuntu patch delta from the upstream
> >>> kernel when the Oneiric kernel opened was ~150 patches.
> >> Yes but if you look closely you will see that most of the patches are
> >> either small or add a new feature that doesn't conflict with the vanilla
> >> kernel.
> >> The 32 patches in the trace-event-semicolon-removal are split per file
> >> they modify and usually modify very few lines. Moreover these patches
> >> are being mainlined as we speak (really short ETA).
> > 
> > Does "really short ETA" imply the patches will land in the 2.6.40
> > kernel?  Are they already in a subsystem maintainer's tree?
> All I can tell you is that, they were posted yesterday and are already
> being acked by the maintainers :
> https://lkml.org/lkml/2011/5/2/350

Yes. So it's not in a maintainer tree yet, but very close (received many
acked-by, strong support from Steven Rostedt). So this part could make
it into 2.6.40.

> 
> >> The lib-ring-buffer patches will enter in the mainline kernel as soon as
> >> we make perf work with it (around september).
> > 
> > How long do you think this work will take?  Which mainline kernel would
> > you anticipate them to land in?
> Probably Mathieu should answer this one.

Probably .41 or .42: push happening during the .40-rc, aiming at the
merge window kicked off by the .40 release, which leads us to be merged
for .41 (if everything goes well), or .42 (pessimistic scenario).

> 
> >> The trace-clock patches are architecture specific and work to mainline
> >> these is in progress also (all the ARM related patches are already in
> >> the Ubuntu kernel IIRC).
> > 
> > Same question, which upstream kernel version do you expect the
> > trace-clock patches to land?  2.6.40?  Also, I don't see any ARM related
> > trace-clock patches in the Ubuntu distro.
> The ARM related trace-clock patches are in Linaro kernel actually.

Yes.

> 
> >>>> The Linaro project already ships with LTTng. For the next Ubuntu
> >>>> release, it would be really interesting to have a LTTng-ready kernel in
> >>>> archive to see if it could become the default kernel in the next LTS.
> >>>> Our mainlining expectations for the biggest patches are around september
> >>>> 2011.
> > 
> > So the reason I keep inquiring about which upstream kernel you
> > anticipate patches to land is because it's a good possibility Oneiric's
> > kernel will at least be a 2.6.40 based kernel.  If a good majority of
> > the patches land in 2.6.40 we can then re-evaluate what's left that
> > needs applied, if any.  In my opinion, I think you effort is best spent
> > getting the patches applied upstream this cycle.

As I described above, I don't think porting Perf to the generic ring
buffer library is realistically going to happen for 2.6.40. But on the
other hand, the generic ring buffer patches are very much independent
from the rest of the kernel, so they would not be a big burden to
support.

Thank you,

Mathieu

> I agree and that is our primary focus, the reason we would like to have
> it "early" in Ubuntu is to make sure it will be in the next LTS.
> 
> The real number of patches to integrate for supported architectures in
> Ubuntu is 43 patches, some of them are really just one liner and most of
> create new files or add new functions.
> Still I understand and agree that integrating out-of-tree patches has a
> cost in maintenance. All I would like to make sure is that when the next
> LTS comes up, we will have a good ground to ease the integration.
> 
> Thanks,
> 
> Julien

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com