[Bug 1384660] Re: Idle rpc traffic with a large number of instances causes failures

[Armando Migliaccio]

This bug is > 365 days without activity. We are unsetting assignee and
milestone and setting status to Incomplete in order to allow its expiry
in 60 days.

If the bug is still valid, then update the bug status.

** Changed in: neutron
       Status: Confirmed => Incomplete

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1384660

Title:
  Idle rpc traffic with a large number of instances causes failures

Status in Ubuntu Cloud Archive:
  Fix Released
Status in neutron:
  Incomplete

Bug description:
  OpenStack Juno (Neutron ML2+OVS/l2pop/neutron security groups), Ubuntu
  14.04

  500 compute node cloud, running 4.5k active instances (can't get it
  any further right now).

  As the number of instances in the cloud increases, the idle loading on
  the neutron-server servers (4 of them all with 4 cores/8 threads and a
  suitable *_worker configuration) increases from nothing to 30;  The db
  call get_port_and_sgs is being serviced around 10 times per second on
  each server at this point. Other things are also happening - I've
  attached the last 1000 lines of the server log with debug enabled.

  The result is that its no longer possible to create new instances, as
  the rpc calls and api thread just don't get onto CPU, resulting in VIF
  plugging timeouts on compute nodes, and ERROR'ed instances.

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: neutron-common 1:2014.2-0ubuntu1~cloud0 [origin: Canonical]
  ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6
  Uname: Linux 3.13.0-35-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.5
  Architecture: amd64
  CrashDB:
   {
                  "impl": "launchpad",
                  "project": "cloud-archive",
                  "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
               }
  Date: Thu Oct 23 10:22:14 2014
  PackageArchitecture: all
  SourcePackage: neutron
  UpgradeStatus: No upgrade log present (probably fresh install)
  modified.conffile..etc.neutron.api.paste.ini: [deleted]
  modified.conffile..etc.neutron.fwaas.driver.ini: [deleted]
  modified.conffile..etc.neutron.l3.agent.ini: [deleted]
  modified.conffile..etc.neutron.neutron.conf: [deleted]
  modified.conffile..etc.neutron.policy.json: [deleted]
  modified.conffile..etc.neutron.rootwrap.conf: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.ipset.firewall.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted]
  modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted]
  modified.conffile..etc.neutron.vpn.agent.ini: [deleted]
  modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted]

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1384660/+subscriptions