[Bug 1169740] Re: rsyslog hangs loading modules

[Louis Bouchard]

Hi,

I had to modify the reproducer script which did trigger "false positive"
(the noonce pattern getting into syslog after the grep) by adding a
sleep 1 after :

if grep $nonce /var/log/hourly/* > /dev/null; then

The new package in -proposed has been running successfully for more than
3 hours, but the original package does not trigger a FAIL from the
script either with the added sleep 1.

I am tempted to mark it 'verification-done' since it has been running
correctly for a long period. Maybe Adam wants to comment ?

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1169740

Title:
  rsyslog hangs loading modules

Status in “rsyslog” package in Ubuntu:
  Fix Released
Status in “rsyslog” source package in Precise:
  Fix Committed
Status in “rsyslog” source package in Quantal:
  Fix Committed
Status in “rsyslog” source package in Raring:
  Fix Released

Bug description:
  [Impact]
  We have rsyslog configured to listen for forwarded logs on TCP and UDP, and to forward logs on to a central log host. On a number of occasions we have observed rsyslog in a hung state on our production machines -- not fowarding logs and not writing them to disk locally. It doesn't happen all the time, so I believe there's some sort of race condition involved.

  I've attached a stack trace taken when the process was hung. I believe
  the issue is the same as the one discussed here: http://www.gossamer-
  threads.com/lists/rsyslog/users/5873 There is a patch attached to that
  thread.

  [Test Case]
  #!/bin/sh

  while [ 1 ]; do
    service rsyslog stop
    sleep 1
    killall -9 rsyslogd > /dev/null 2>&1
    service rsyslog start
    sleep 1
    nonce=$(date '+%s')
    logger $nonce
    if grep $nonce /var/log/hourly/* > /dev/null; then
      echo found nonce $nonce
    else
      echo FAIL
      exit 1
    fi
  done

  [Regression Potential]
  This is a fix for a deadlock issue it is pretty difficult to determine if fixing a race condition would impact anything else, however, testing from the community hasn't raised any concerns.

  [Additional Info]

  Ubuntu 12.04.1 LTS x86_64
  rsyslog 5.8.6-1ubuntu8

  Here is the (obscured) section of our rsyslog config that I believe is
  causing the issue.

  $ActionQueueType LinkedList
  $ActionQueueFileName central_syslog
  $ActionQueueMaxDiskSpace 256M
  $ActionResumeRetryCount -1
  $ActionQueueSaveOnShutdown on
  $ActionSendTCPRebindInterval 100000
  *.* @@central_syslog.domain:514

  $ModLoad imudp
  $UDPServerRun 514

  $ModLoad imtcp
  $InputTCPServerRun 514

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1169740/+subscriptions