Integrating LTTng in Ubuntu

Wed May 4 14:09:47 UTC 2011

On 05/03/2011 04:04 PM, Julien Desfossez wrote:
> On 11-05-03 10:44 AM, Leann Ogasawara wrote:
>> On Tue, 2011-05-03 at 00:09 -0400, Julien Desfossez wrote:
>>> On 11-05-02 04:55 PM, Leann Ogasawara wrote:
>>>> On Mon, 2011-05-02 at 14:05 -0400, Julien Desfossez wrote:
>>>>> Hello,
>>>>>
>>>>> LTTng, the Linux Trace Toolkit Next Generation, is a project that aims
>>>>> at producing a highly efficient full system tracing solution. It is
>>>>> composed of several components to allow tracing of the kernel, of
>>>>> userspace, trace viewing and analysis and trace streaming.
>>>>>
>>>>> Ubuntu is more and more popular as a server-grade distribution and
>>>>> having a comprehensive tracing infrastructure built into the kernel
>>>>> would allow system administrators to investigate problems as they appear
>>>>> without patching the kernel and rebooting the server.
>>>>>
>>>>> For example, with LTTng you know really quickly what is the I/O
>>>>> bandwidth (network or disk) associated with a particular process, and in
>>>>> this process which file is the culprit for slowing down your whole
>>>>> system. LTTng can also be used as a highly efficient userspace tracer
>>>>> with UST. When the user-space and kernel-space traces are combined,
>>>>> application developpers and sysadmins have a very detailed overview of
>>>>> what is happening on their system.
>>>>>
>>>>> Over the last months, the size of the LTTng kernel patchset has been
>>>>> considerably shrinked for Linux distributions by removing all
>>>>> LTTng-specific instrumentation from the tree.
>>>>>
>>>>> You can find the latest patchset here [1].
>>>>>
>>>>> Please note that the patches within this patchset are elected as the
>>>>> first items from the LTTng project to eventually get into the mainline
>>>>> Linux kernel. This includes TRACE_EVENT() changes, the LTTng trace
>>>>> clocks and the Generic Ring Buffer Library. Please take into account
>>>>> that most of those patches are really small patches that have been split
>>>>> to facilitate mainlining by addressing them to the appropriate maintainers.
>>>>>
>>>>> Also, you will notice that most of the bigger patches are actually new
>>>>> functionalities that won't conflict with existing code base, because
>>>>> they create files in new directories of the kernel tree.
>>>>>
>>>>> The upcoming UDS would be a good timing to discuss the integration of
>>>>> the new LTTng as an efficient tracer especially for the server flavour
>>>>> of the Ubuntu kernel.
>>>>>
>>>>> I submitted a blueprint [2] and following the discussion with Leann
>>>>> Ogasawara I'm asking the Ubuntu Kernel-team their feedbacks and opinions
>>>>> about making a full fledged blueprint/spec. I will be in Budapest next week.
>>>>
>>>> One of my concerns is the maintenance burden this brings upon the Ubuntu
>>>> Kernel Team.  The LTTng patch set looks like it's 86 patches at the
>>>> moment.  In comparison, our entire Ubuntu patch delta from the upstream
>>>> kernel when the Oneiric kernel opened was ~150 patches.
>>> Yes but if you look closely you will see that most of the patches are
>>> either small or add a new feature that doesn't conflict with the vanilla
>>> kernel.
>>> The 32 patches in the trace-event-semicolon-removal are split per file
>>> they modify and usually modify very few lines. Moreover these patches
>>> are being mainlined as we speak (really short ETA).
>>
>> Does "really short ETA" imply the patches will land in the 2.6.40
>> kernel?  Are they already in a subsystem maintainer's tree?
> All I can tell you is that, they were posted yesterday and are already
> being acked by the maintainers :
> https://lkml.org/lkml/2011/5/2/350
>
>>> The lib-ring-buffer patches will enter in the mainline kernel as soon as
>>> we make perf work with it (around september).
>>
>> How long do you think this work will take?  Which mainline kernel would
>> you anticipate them to land in?
> Probably Mathieu should answer this one.
>
>>> The trace-clock patches are architecture specific and work to mainline
>>> these is in progress also (all the ARM related patches are already in
>>> the Ubuntu kernel IIRC).
>>
>> Same question, which upstream kernel version do you expect the
>> trace-clock patches to land?  2.6.40?  Also, I don't see any ARM related
>> trace-clock patches in the Ubuntu distro.
> The ARM related trace-clock patches are in Linaro kernel actually.
>
>>>>> The Linaro project already ships with LTTng. For the next Ubuntu
>>>>> release, it would be really interesting to have a LTTng-ready kernel in
>>>>> archive to see if it could become the default kernel in the next LTS.
>>>>> Our mainlining expectations for the biggest patches are around september
>>>>> 2011.
>>
>> So the reason I keep inquiring about which upstream kernel you
>> anticipate patches to land is because it's a good possibility Oneiric's
>> kernel will at least be a 2.6.40 based kernel.  If a good majority of
>> the patches land in 2.6.40 we can then re-evaluate what's left that
>> needs applied, if any.  In my opinion, I think you effort is best spent
>> getting the patches applied upstream this cycle.
> I agree and that is our primary focus, the reason we would like to have
> it "early" in Ubuntu is to make sure it will be in the next LTS.
>
> The real number of patches to integrate for supported architectures in
> Ubuntu is 43 patches, some of them are really just one liner and most of
> create new files or add new functions.
> Still I understand and agree that integrating out-of-tree patches has a
> cost in maintenance. All I would like to make sure is that when the next
> LTS comes up, we will have a good ground to ease the integration.

Yes it is very expensive. We carry a high maintenance overhead for 
supporting non-LTS releases & 3 active LTS releases. Every patch that is 
not in mainline has to be carried and fixed up quite frequently 
depending where in the tree it is. Additionaly we have to take extra 
care with CVEs to make sure we are non vulnerable due to extra patches. 
Our policy is only to do it for items that directly benefit Ubuntu where 
we are willing to pay the extra cost aka aufs used for live cd's.

Thanks
-- 
Pete Graner          <pgraner at canonical.com>
Manager
Ubuntu Kernel Team
Canonical Ltd.       http://www.canonical.com/