dbus daemon/messagebus 100% CPU usage

Thu Sep 23 19:38:26 BST 2010

Hello,

I just had this happen on my brand new 10.04 LTSP system for the first time
today. I'm not sure how many users I had logged in at the time.

Users could not log in nor could I ssh or log directly into the server when
this was happening.  Looks like the problem went away when users logged out
at lunch.

-Nick

On Tue, Sep 21, 2010 at 8:26 AM, David Hopkins <dahopkins429 at gmail.com>wrote:

> Ok .. I edited the limits.conf file and added what I thought was the
> correct information e.g.
>
> * hard nofile 65535
>
> and then also edited the pam file as well (following
> http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/ ),
> rebooted the server and ... today the problem has returned.  I am at
> wits end.  An identical system which only has 36 users does not have
> any issues at all. However, on this system, once we reach
> approximately 50 users, we start to have issues with messagebus
> (dbus-daemon).  I can move users to another server but that isn't
> exactly solving the issue so much as avoiding it and means adding
> another server. As the load is only around 2-3 on a dual quad-core
> box, load isn't the issue ... it some setting and I really think it is
> nofile
>
> I guess I'll add another server to the mix in the interim just to get
> around this problem but I'd prefer to actually solve it.
>
> Sincerely,
> Dave Hopkins
>
> On Fri, Sep 17, 2010 at 9:10 PM, David Hopkins <dahopkins429 at gmail.com>
> wrote:
> > I tried this but not sure how to tell if it actually set nofiles for
> > dbus to the desired value.  A shell still reports 1024 as the value
> > though that is just for that user and not for messagebus or
> > dbus-daemon?
> >
> > On Fri, Sep 17, 2010 at 6:20 PM, Jordan Erickson
> > <jerickson at logicalnetworking.net> wrote:
> >> Looks like you can add the line "ulimit -n 65535" to /etc/default/dbus
> >>
> >> ... :)
> >>
> >> https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/381063/comments/4
> >>
> >>
> >> Cheers,
> >> Jordan
> >>
> >>
> >> David Hopkins wrote:
> >>> It appears that the issue is related to this bug:
> >>>
> >>> https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/381063
> >>>
> >>> So, how do I set ulimit -n 8192
> >>>
> >>> for example to increase the open files limit?  Do I need to edit the
> >>> /etc/init/dbus.conf file and add this command somewhere? Or is there
> >>> some other location that I can make certain that the limit is changed
> >>> from 1024 to a larger number?
> >>>
> >>> I would think this bug would affect more than just me as it seems to
> >>> be an issue for ltsp in general.
> >>>
> >>> Sincerely,
> >>> Dave Hopkins
> >>>
> >>>
> >>> On Thu, Sep 16, 2010 at 1:42 PM, David Hopkins <dahopkins429 at gmail.com>
> wrote:
> >>>> Jordan,
> >>>>
> >>>> Thanks for the link but I don't think it is that issue as I'm not
> >>>> getting error messages in the log files.  Running dpkg
> >>>> --get-selections shows that the two systems have exactly the same
> >>>> packages installed.  I have between 40 and 50 users logged onto the
> >>>> server at any given time.  It is rather random as the system can run
> >>>> fine for a day or so and then suddenly have this issue.
> >>>>
> >>>> I have already rebuilt the server once a few weeks ago. That was when
> >>>> I made certain that is was a 'clone' of the other server.  I used the
> >>>> --get-selections and --set-selections to ensure that apt installed
> >>>> only those packages.
> >>>>
> >>>> The only clue I have at present is that npviewer.bin (flash?) will
> >>>> segfault at times though the system doesn't misbehave when this
> >>>> happens. It is probably just one line in a log file that I haven't
> >>>> located which the clue I need, but if I can't resolve it quickly, I'll
> >>>> probably have to try upgrading the systems to Lucid (10.04) although
> >>>> from some traffic on the lists, it seems lucid comes with its own set
> >>>> of issues.
> >>>>
> >>>> Thanks!
> >>>> Dave ...
> >>>>
> >>>> On Thu, Sep 16, 2010 at 11:24 AM, Jordan Erickson
> >>>> <jerickson at logicalnetworking.net> wrote:
> >>>>> Hi David,
> >>>>>
> >>>>> If 1 of 2 servers is experiencing this issue, and nobody else has
> chimed
> >>>>> in yet to say they're having the same problem, I'd have to assume
> it's
> >>>>> something very particular to your one server's setup.. A couple of
> >>>>> questions:
> >>>>>
> >>>>> - Is this happening with specific users, only before/after login,
> ...?
> >>>>> - Have you checked logs on the terminals themselves?
> >>>>> - Just a shot in the dark (google in the dark?) but have you seen
> >>>>> comments 11 & 12 at
> >>>>> https://bugs.launchpad.net/debian/+source/pulseaudio/+bug/390528 ?
> >>>>>
> >>>>> I'm sure there's a solution to your problem. The important thing IMHO
> is
> >>>>> to do as much research on your issue as possible and not let the
> school
> >>>>> director think that it's unsolvable. Worst case scenario you could
> >>>>> always wipe/re-install the system. Going back to an unsupported LTSP
> 4.2
> >>>>> probably isn't your best option.
> >>>>>
> >>>>>
> >>>>> Cheers,
> >>>>> Jordan
> >>>>>
> >>>>>
> >>>>> On 09/16/2010 05:12 AM, David Hopkins wrote:
> >>>>>> I am still having issues with the dbus-daemon randomly consuming
> 100%
> >>>>>> of the cpu on my servers which renders them unusable.  There doesn't
> >>>>>> appear to be any messages in the logs but the school director has
> now
> >>>>>> stopped by and asked if it wouldn't be simpler to go back to the
> RHEL5
> >>>>>> with LTSP4.2 instead of staying with Ubuntu.  He is very supportive
> of
> >>>>>> using LTSP and for him to ask this is a serious concern.
> >>>>>>
> >>>>>> Does anyone have any ideas about why the dbus-daemon would suddenly
> >>>>>> spike to 100% cpu usage and hang the system?  The load average can
> be
> >>>>>> very low (4-5 on a dual-quad) and yet the system is completely
> >>>>>> nonresponsive?
> >>>>>>
> >>>>>> Sincerely,
> >>>>>> Dave Hopkins
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Sep 9, 2010 at 3:01 PM, David Hopkins <
> dahopkins429 at gmail.com> wrote:
> >>>>>>> Has anyone noticed dbus (messagebus) using 100% of a CPU on an LTSP
> >>>>>>> server?  I have two identical servers both hardware and software
> >>>>>>> (Karmic 9.10, comparing output from dpkg --get-selections).
> However,
> >>>>>>> one server runs flawlessly, the other shows dbus_daemon using 100%
> of
> >>>>>>> a CPU. Also, memory usage slowly climbs until the system begins
> using
> >>>>>>> virtual memory and everything then comes to a halt.  Both systems
> >>>>>>> connect to the same brand of switch, both connect to a KVM.  The
> only
> >>>>>>> differences are in the client systems that they serve. One servers
> >>>>>>> GX150/GX240s while the other (the misbehaving system) serves
> >>>>>>> GX50/GX110/GX150's.  I'm in the process of replacing the GX50
> systems
> >>>>>>> and as many GX110's as possible, but curious if anyone has noticed
> >>>>>>> this issue.
> >>>>>>>
> >>>>>>> Sincerely,
> >>>>>>> Dave Hopkins
> >>>>>>> Newark Charter School
> >>>>>>> Newark Delaware
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>> Jordan Erickson - LNS
> >>>>> (707) 636-5678  - http://logicalnetworking.net
> >>>>>
> >>>>> --
> >>>>> edubuntu-users mailing list
> >>>>> edubuntu-users at lists.ubuntu.com
> >>>>> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
> >>>>>
> >>
> >>
> >> --
> >> Jordan Erickson - LNS
> >> (707) 636-5678  - http://logicalnetworking.net
> >>
> >> --
> >> edubuntu-users mailing list
> >> edubuntu-users at lists.ubuntu.com
> >> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
> >>
> >
>
> --
> edubuntu-users mailing list
> edubuntu-users at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/edubuntu-users
>

-- 
Nick Fenger
-Information Technology

Trillium Charter School
5420 N. Interstate Ave
Portland, OR 97217
(503) 285-3833
http://www.trilliumcharterschool.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/edubuntu-users/attachments/20100923/5250e6c6/attachment.htm 