[Bug 1683076] Re: Swift-storage dies if rsyslog is stopped

Nivedita Singhvi nivedita.singhvi at canonical.com
Fri Feb 16 18:02:57 UTC 2018

Was the behavior of rsyslog prior to this fix as described in 1683075,
by any chance? Did the resolution wrt the storage issue affect the rsyslog
frequent stoppage? I'm trying to determine if 1683075 is still an issue..

You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to swift in Ubuntu.

  Swift-storage dies if rsyslog is stopped

Status in OpenStack swift-storage charm:
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive icehouse series:
  Won't Fix
Status in Ubuntu Cloud Archive kilo series:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Fix Released
Status in Ubuntu Cloud Archive newton series:
  Fix Released
Status in swift package in Ubuntu:
Status in swift source package in Trusty:
  Fix Released
Status in swift source package in Xenial:
  Fix Released

Bug description:
  Trusty, Mitaka, Juju 1.25.11

  We have a cloud where swift replicators are constantly falling over on
  2 nodes.  This occurs whenever rsyslog restarts, as in


  rsyslog restarts are unfortunately frequent right now, due to

  Nodes are landscape managed and up to date but still exhibit the

  Not much from running swift in verbose.

  sosreports are uploading to https://private-


   * Stopping rsyslog causes swift daemons to crash due to overflowing
  the call stack when attempting to write an entry to the logging
  subsystem and the attempt to write to /dev/log fails. When rsyslog
  stops, the /dev/log socket is unavailable and results in an exception.
  The swift logging code attempts to log the resultant error, which
  again results in an exception. This continues until the stack is
  overflowed and the swift daemons crash. When the swift daemons crash,
  the object, container and account data are not able to be replicated
  to other storage nodes in the system, which affects the data integrity
  of the data being written to the system.

   * The patch should be backported to stable releases in order to
  ensure that the data integrity of objects, accounts, and containers
  within Swift are not adversely affected due to failed logging

   * The uploaded patches fix the bug by only attempting to log an entry
  to the logging subsystem if the current call stack does not include an
  attempt to write to the logs. If the current call stack includes an
  attempt to log to the logging subsystem, the log message is dropped
  avoiding the recursion.

  [Test Case]

   * Install swift storage cluster
   * Log into one of the swift storage nodes
   * Ensure the swift-{object,account,container}-replicator processes are running
   * Stop the rsyslog service
   * Wait a minute
   * Observe the swift-{object,account,container}-replicator processes are no longer running

  [Regression Potential]

   * This affects the logging capabilities provided by the Swift code.
  Possible regressions could occur in almost any subsystem, since the
  logging is universal throughout the code base and could result in lost
  log entries in the best regression scenario and possible crashing of
  swift daemons in the worst case scenario. The regression potential is
  mitigated by the fact that this patch has already been included
  upstream for over a year now and no regressions have been reported
  against this code since.

  [Other Info]

   * /dev/log is not provided by the rsyslog daemon in Xenial, but this
  patch still applies in that any persistent exception encountered when
  writing to /dev/log will cause the call stack to overflow and crash
  the swift daemons.

To manage notifications about this bug go to:

More information about the Ubuntu-openstack-bugs mailing list