[Bug 1388077] Re: Parallel periodic instance power state reporting from compute nodes has high impact on conductors and message broker

[Markus Zoeller (markus_z)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY


** Changed in: nova
   Importance: Medium => Undecided

** Changed in: nova
       Status: Confirmed => Expired

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1388077

Title:
  Parallel periodic instance power state reporting from compute nodes
  has high impact on conductors and message broker

Status in OpenStack Compute (nova):
  Expired
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Utopic:
  Fix Released
Status in nova source package in Vivid:
  Fix Released

Bug description:
  Environment: OpenStack Juno release/Ubuntu 14.04/480 compute nodes/8
  cloud controllers/40,000 instances +

  The change made in:

  https://github.com/openstack/nova/commit/baabab45e0ae0e9e35872cae77eb04bdb5ee0545

  switches power state reporting from being a serial process for each
  instance on a hypervisor to being a parallel thread for every
  instance; for clouds running high instance counts, this has quite an
  impact on the conductor processes as they try to deal with N instance
  refresh calls in parallel where N is the number of instances running
  on the cloud.

  It might be better to throttle this to a configurable parallel level
  so that period RPC load can be managed effectively in a larger cloud,
  or to continue todo this process in series but outside of the main
  thread.

  The net result of this activity is that it places increase demands on
  the message broker, which has to deal with more parallel connections,
  and the conductors as they try to consume all of the RPC requests; if
  the message broker hits its memory high water mark it will stop
  publishers publishing any more messages until the memory usage drops
  below the high water mark again - this might not be achievable if all
  conductor processes are tied up with existing RPC calls try to send
  replies, resulting in a message broker lockup and collapse of all RPC
  in the cloud.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1388077/+subscriptions