[Bug 1683076] Re: Swift-storage dies if rsyslog is stopped

Robie Basak 1683076 at bugs.launchpad.net
Wed Jul 12 14:10:49 UTC 2017


15:07 <rbasak> wolsen: around? You need a sponsor for bug 1683076 I
think?

15:08 <rbasak> wolsen: I suggest three minor changes to your trusty
debdiff:
https://git.launchpad.net/~racb/ubuntu/+source/swift/log/?h=lp1683076-trusty

15:09 <rbasak> The dep3 changes apply to your xenial debdiff too I
think. But I'll accept without if you wish.

Can you find a sponsor for the Trusty debdiff please, and then I can
accept the SRU?

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to swift in Ubuntu.
https://bugs.launchpad.net/bugs/1683076

Title:
  Swift-storage dies if rsyslog is stopped

Status in OpenStack swift-storage charm:
  Invalid
Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive icehouse series:
  Won't Fix
Status in Ubuntu Cloud Archive kilo series:
  New
Status in Ubuntu Cloud Archive mitaka series:
  New
Status in Ubuntu Cloud Archive newton series:
  Fix Released
Status in swift package in Ubuntu:
  Invalid
Status in swift source package in Trusty:
  New
Status in swift source package in Xenial:
  New

Bug description:
  Trusty, Mitaka, Juju 1.25.11

  We have a cloud where swift replicators are constantly falling over on
  2 nodes.  This occurs whenever rsyslog restarts, as in

  https://bugs.launchpad.net/swift/+bug/1094230
  https://review.openstack.org/#/c/24871
  https://bugs.python.org/issue15179

  rsyslog restarts are unfortunately frequent right now, due to
  https://bugs.launchpad.net/juju-core/+bug/1683075

  Nodes are landscape managed and up to date but still exhibit the
  failure.

  Not much from running swift in verbose.
  https://pastebin.canonical.com/185609/

  sosreports are uploading to https://private-
  fileshare.canonical.com/~jillr/sf00137831/

  [Impact]

   * Stopping rsyslog causes swift daemons to crash due to overflowing
  the call stack when attempting to write an entry to the logging
  subsystem and the attempt to write to /dev/log fails. When rsyslog
  stops, the /dev/log socket is unavailable and results in an exception.
  The swift logging code attempts to log the resultant error, which
  again results in an exception. This continues until the stack is
  overflowed and the swift daemons crash. When the swift daemons crash,
  the object, container and account data are not able to be replicated
  to other storage nodes in the system, which affects the data integrity
  of the data being written to the system.

   * The patch should be backported to stable releases in order to
  ensure that the data integrity of objects, accounts, and containers
  within Swift are not adversely affected due to failed logging
  subsystems.

   * The uploaded patches fix the bug by only attempting to log an entry
  to the logging subsystem if the current call stack does not include an
  attempt to write to the logs. If the current call stack includes an
  attempt to log to the logging subsystem, the log message is dropped
  avoiding the recursion.

  [Test Case]

   * Install swift storage cluster
   * Log into one of the swift storage nodes
   * Ensure the swift-{object,account,container}-replicator processes are running
   * Stop the rsyslog service
   * Wait a minute
   * Observe the swift-{object,account,container}-replicator processes are no longer running

  [Regression Potential]

   * This affects the logging capabilities provided by the Swift code.
  Possible regressions could occur in almost any subsystem, since the
  logging is universal throughout the code base and could result in lost
  log entries in the best regression scenario and possible crashing of
  swift daemons in the worst case scenario. The regression potential is
  mitigated by the fact that this patch has already been included
  upstream for over a year now and no regressions have been reported
  against this code since.

  [Other Info]

   * /dev/log is not provided by the rsyslog daemon in Xenial, but this
  patch still applies in that any persistent exception encountered when
  writing to /dev/log will cause the call stack to overflow and crash
  the swift daemons.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-swift-storage/+bug/1683076/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list