dbus daemon/messagebus 100% CPU usage

Thu Sep 23 19:55:17 BST 2010

The fix is simple once you know what file to modify.  See
https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/381063

Thanks are due to Jakob on the LTSP mailing list.

I added the

limit nofile 10000 10000

line just before the expect fork line in dbus.conf

Dave ...

On Thu, Sep 23, 2010 at 2:38 PM, Nick Fenger
<nick at trilliumcharterschool.org> wrote:
> Hello,
> I just had this happen on my brand new 10.04 LTSP system for the first time
> today. I'm not sure how many users I had logged in at the time.
> Users could not log in nor could I ssh or log directly into the server when
> this was happening.  Looks like the problem went away when users logged out
> at lunch.
> -Nick
>
> On Tue, Sep 21, 2010 at 8:26 AM, David Hopkins <dahopkins429 at gmail.com>
> wrote:
>>
>> Ok .. I edited the limits.conf file and added what I thought was the
>> correct information e.g.
>>
>> * hard nofile 65535
>>
>> and then also edited the pam file as well (following
>> http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/ ),
>> rebooted the server and ... today the problem has returned.  I am at
>> wits end.  An identical system which only has 36 users does not have
>> any issues at all. However, on this system, once we reach
>> approximately 50 users, we start to have issues with messagebus
>> (dbus-daemon).  I can move users to another server but that isn't
>> exactly solving the issue so much as avoiding it and means adding
>> another server. As the load is only around 2-3 on a dual quad-core
>> box, load isn't the issue ... it some setting and I really think it is
>> nofile
>>
>> I guess I'll add another server to the mix in the interim just to get
>> around this problem but I'd prefer to actually solve it.
>>
>> Sincerely,
>> Dave Hopkins
>>
>> On Fri, Sep 17, 2010 at 9:10 PM, David Hopkins <dahopkins429 at gmail.com>
>> wrote:
>> > I tried this but not sure how to tell if it actually set nofiles for
>> > dbus to the desired value.  A shell still reports 1024 as the value
>> > though that is just for that user and not for messagebus or
>> > dbus-daemon?
>> >
>> > On Fri, Sep 17, 2010 at 6:20 PM, Jordan Erickson
>> > <jerickson at logicalnetworking.net> wrote:
>> >> Looks like you can add the line "ulimit -n 65535" to /etc/default/dbus
>> >>
>> >> ... :)
>> >>
>> >> https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/381063/comments/4
>> >>
>> >>
>> >> Cheers,
>> >> Jordan
>> >>
>> >>
>> >> David Hopkins wrote:
>> >>> It appears that the issue is related to this bug:
>> >>>
>> >>> https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/381063
>> >>>
>> >>> So, how do I set ulimit -n 8192
>> >>>
>> >>> for example to increase the open files limit?  Do I need to edit the
>> >>> /etc/init/dbus.conf file and add this command somewhere? Or is there
>> >>> some other location that I can make certain that the limit is changed
>> >>> from 1024 to a larger number?
>> >>>
>> >>> I would think this bug would affect more than just me as it seems to
>> >>> be an issue for ltsp in general.
>> >>>
>> >>> Sincerely,
>> >>> Dave Hopkins
>> >>>
>> >>>
>> >>> On Thu, Sep 16, 2010 at 1:42 PM, David Hopkins
>> >>> <dahopkins429 at gmail.com> wrote:
>> >>>> Jordan,
>> >>>>
>> >>>> Thanks for the link but I don't think it is that issue as I'm not
>> >>>> getting error messages in the log files.  Running dpkg
>> >>>> --get-selections shows that the two systems have exactly the same
>> >>>> packages installed.  I have between 40 and 50 users logged onto the
>> >>>> server at any given time.  It is rather random as the system can run
>> >>>> fine for a day or so and then suddenly have this issue.
>> >>>>
>> >>>> I have already rebuilt the server once a few weeks ago. That was when
>> >>>> I made certain that is was a 'clone' of the other server.  I used the
>> >>>> --get-selections and --set-selections to ensure that apt installed
>> >>>> only those packages.
>> >>>>
>> >>>> The only clue I have at present is that npviewer.bin (flash?) will
>> >>>> segfault at times though the system doesn't misbehave when this
>> >>>> happens. It is probably just one line in a log file that I haven't
>> >>>> located which the clue I need, but if I can't resolve it quickly,
>> >>>> I'll
>> >>>> probably have to try upgrading the systems to Lucid (10.04) although
>> >>>> from some traffic on the lists, it seems lucid comes with its own set
>> >>>> of issues.
>> >>>>
>> >>>> Thanks!
>> >>>> Dave ...
>> >>>>
>> >>>> On Thu, Sep 16, 2010 at 11:24 AM, Jordan Erickson
>> >>>> <jerickson at logicalnetworking.net> wrote:
>> >>>>> Hi David,
>> >>>>>
>> >>>>> If 1 of 2 servers is experiencing this issue, and nobody else has
>> >>>>> chimed
>> >>>>> in yet to say they're having the same problem, I'd have to assume
>> >>>>> it's
>> >>>>> something very particular to your one server's setup.. A couple of
>> >>>>> questions:
>> >>>>>
>> >>>>> - Is this happening with specific users, only before/after login,
>> >>>>> ...?
>> >>>>> - Have you checked logs on the terminals themselves?
>> >>>>> - Just a shot in the dark (google in the dark?) but have you seen
>> >>>>> comments 11 & 12 at
>> >>>>> https://bugs.launchpad.net/debian/+source/pulseaudio/+bug/390528 ?
>> >>>>>
>> >>>>> I'm sure there's a solution to your problem. The important thing
>> >>>>> IMHO is
>> >>>>> to do as much research on your issue as possible and not let the
>> >>>>> school
>> >>>>> director think that it's unsolvable. Worst case scenario you could
>> >>>>> always wipe/re-install the system. Going back to an unsupported LTSP
>> >>>>> 4.2
>> >>>>> probably isn't your best option.
>> >>>>>
>> >>>>>
>> >>>>> Cheers,
>> >>>>> Jordan
>> >>>>>
>> >>>>>
>> >>>>> On 09/16/2010 05:12 AM, David Hopkins wrote:
>> >>>>>> I am still having issues with the dbus-daemon randomly consuming
>> >>>>>> 100%
>> >>>>>> of the cpu on my servers which renders them unusable.  There
>> >>>>>> doesn't
>> >>>>>> appear to be any messages in the logs but the school director has
>> >>>>>> now
>> >>>>>> stopped by and asked if it wouldn't be simpler to go back to the
>> >>>>>> RHEL5
>> >>>>>> with LTSP4.2 instead of staying with Ubuntu.  He is very supportive
>> >>>>>> of
>> >>>>>> using LTSP and for him to ask this is a serious concern.
>> >>>>>>
>> >>>>>> Does anyone have any ideas about why the dbus-daemon would suddenly
>> >>>>>> spike to 100% cpu usage and hang the system?  The load average can
>> >>>>>> be
>> >>>>>> very low (4-5 on a dual-quad) and yet the system is completely
>> >>>>>> nonresponsive?
>> >>>>>>
>> >>>>>> Sincerely,
>> >>>>>> Dave Hopkins
>> >>>>>>
>> >>>>>>
>> >>>>>> On Thu, Sep 9, 2010 at 3:01 PM, David Hopkins
>> >>>>>> <dahopkins429 at gmail.com> wrote:
>> >>>>>>> Has anyone noticed dbus (messagebus) using 100% of a CPU on an
>> >>>>>>> LTSP
>> >>>>>>> server?  I have two identical servers both hardware and software
>> >>>>>>> (Karmic 9.10, comparing output from dpkg --get-selections).
>> >>>>>>> However,
>> >>>>>>> one server runs flawlessly, the other shows dbus_daemon using 100%
>> >>>>>>> of
>> >>>>>>> a CPU. Also, memory usage slowly climbs until the system begins
>> >>>>>>> using
>> >>>>>>> virtual memory and everything then comes to a halt.  Both systems
>> >>>>>>> connect to the same brand of switch, both connect to a KVM.  The
>> >>>>>>> only
>> >>>>>>> differences are in the client systems that they serve. One servers
>> >>>>>>> GX150/GX240s while the other (the misbehaving system) serves
>> >>>>>>> GX50/GX110/GX150's.  I'm in the process of replacing the GX50
>> >>>>>>> systems
>> >>>>>>> and as many GX110's as possible, but curious if anyone has noticed
>> >>>>>>> this issue.
>> >>>>>>>
>> >>>>>>> Sincerely,
>> >>>>>>> Dave Hopkins
>> >>>>>>> Newark Charter School
>> >>>>>>> Newark Delaware
>> >>>>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Jordan Erickson - LNS
>> >>>>> (707) 636-5678  - http://logicalnetworking.net
>> >>>>>
>> >>>>> --
>> >>>>> edubuntu-users mailing list
>> >>>>> edubuntu-users at lists.ubuntu.com
>> >>>>> Modify settings or unsubscribe at:
>> >>>>> https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
>> >>>>>
>> >>
>> >>
>> >> --
>> >> Jordan Erickson - LNS
>> >> (707) 636-5678  - http://logicalnetworking.net
>> >>
>> >> --
>> >> edubuntu-users mailing list
>> >> edubuntu-users at lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> >> https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
>> >>
>> >
>>
>> --
>> edubuntu-users mailing list
>> edubuntu-users at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
>
>
>
> --
> Nick Fenger
> -Information Technology
>
> Trillium Charter School
> 5420 N. Interstate Ave
> Portland, OR 97217
> (503) 285-3833
> http://www.trilliumcharterschool.org
>
>
>
>