[Bug 1169740] Re: rsyslog hangs loading modules

[Adam Stokes]

Thanks for all your help Jon. Since raring is being released next week I
can't say for sure when we should expect this into -proposed. I will
bring this up in my next meeting (Wednesday) if I am unable to get at
least the Precise SRU uploaded by then.

Thanks again
Adam

** Description changed:

  [Impact]
  We have rsyslog configured to listen for forwarded logs on TCP and UDP, and to forward logs on to a central log host. On a number of occasions we have observed rsyslog in a hung state on our production machines -- not fowarding logs and not writing them to disk locally. It doesn't happen all the time, so I believe there's some sort of race condition involved.
  
  I've attached a stack trace taken when the process was hung. I believe
  the issue is the same as the one discussed here: http://www.gossamer-
  threads.com/lists/rsyslog/users/5873 There is a patch attached to that
  thread.
  
  [Test Case]
- << TBD >>
+ #!/bin/sh
+ 
+ while [ 1 ]; do
+   service rsyslog stop
+   sleep 1
+   killall -9 rsyslogd > /dev/null 2>&1
+   service rsyslog start
+   sleep 1
+   nonce=$(date '+%s')
+   logger $nonce
+   if grep $nonce /var/log/hourly/* > /dev/null; then
+     echo found nonce $nonce
+   else
+     echo FAIL
+     exit 1
+   fi
+ done
  
  [Regression Potential]
  This is a fix for a deadlock issue it is pretty difficult to determine if fixing a race condition would impact anything else, however, testing from the community hasn't raised any concerns.
  
+ [Additional Info]
+ 
  Ubuntu 12.04.1 LTS x86_64
  rsyslog 5.8.6-1ubuntu8
+ 
+ Here is the (obscured) section of our rsyslog config that I believe is
+ causing the issue.
+ 
+ $ActionQueueType LinkedList
+ $ActionQueueFileName central_syslog
+ $ActionQueueMaxDiskSpace 256M
+ $ActionResumeRetryCount -1
+ $ActionQueueSaveOnShutdown on
+ $ActionSendTCPRebindInterval 100000
+ *.* @@central_syslog.domain:514
+ 
+ $ModLoad imudp
+ $UDPServerRun 514
+ 
+ $ModLoad imtcp
+ $InputTCPServerRun 514

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1169740

Title:
  rsyslog hangs loading modules

Status in “rsyslog” package in Ubuntu:
  Confirmed
Status in “rsyslog” source package in Precise:
  New
Status in “rsyslog” source package in Quantal:
  New
Status in “rsyslog” source package in Raring:
  Confirmed

Bug description:
  [Impact]
  We have rsyslog configured to listen for forwarded logs on TCP and UDP, and to forward logs on to a central log host. On a number of occasions we have observed rsyslog in a hung state on our production machines -- not fowarding logs and not writing them to disk locally. It doesn't happen all the time, so I believe there's some sort of race condition involved.

  I've attached a stack trace taken when the process was hung. I believe
  the issue is the same as the one discussed here: http://www.gossamer-
  threads.com/lists/rsyslog/users/5873 There is a patch attached to that
  thread.

  [Test Case]
  #!/bin/sh

  while [ 1 ]; do
    service rsyslog stop
    sleep 1
    killall -9 rsyslogd > /dev/null 2>&1
    service rsyslog start
    sleep 1
    nonce=$(date '+%s')
    logger $nonce
    if grep $nonce /var/log/hourly/* > /dev/null; then
      echo found nonce $nonce
    else
      echo FAIL
      exit 1
    fi
  done

  [Regression Potential]
  This is a fix for a deadlock issue it is pretty difficult to determine if fixing a race condition would impact anything else, however, testing from the community hasn't raised any concerns.

  [Additional Info]

  Ubuntu 12.04.1 LTS x86_64
  rsyslog 5.8.6-1ubuntu8

  Here is the (obscured) section of our rsyslog config that I believe is
  causing the issue.

  $ActionQueueType LinkedList
  $ActionQueueFileName central_syslog
  $ActionQueueMaxDiskSpace 256M
  $ActionResumeRetryCount -1
  $ActionQueueSaveOnShutdown on
  $ActionSendTCPRebindInterval 100000
  *.* @@central_syslog.domain:514

  $ModLoad imudp
  $UDPServerRun 514

  $ModLoad imtcp
  $InputTCPServerRun 514

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1169740/+subscriptions