[Bug 1477077] [NEW] Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10 stable/kilo

Preethi Dsilva preethi.dsilva at hp.com
Wed Jul 22 10:30:09 UTC 2015 

Public bug reported:

OpenStack stable/kilo release, Ubuntu 14.04 devstack setup .while
multiple calls are generated during tempest run(tempest.api.networks) in
HP networking CI setup we see queuePool limit related exception which
results in tempest failure.

tempest tests that are run include
cd /opt/stack/tempest; testr init ; ./tools/pretty_tox_serial.sh    \"  tempest.api.network.test_networks tempest.api.network.test_extensions tempest.api.network.test_allowed_address_pair tempest.api.network.test_extra_dhcp_options tempest.api.network.test_floating_ips_negative tempest.api.network.test_floating_ips tempest.api.network.test_networks_negative tempest.api.network.test_networks tempest.api.network.test_ports tempest.api.network.test_routers_negative tempest.api.network.test_routers tempest.api.network.test_security_groups_negative tempest.api.network.test_security_groups tempest.api.network.test_service_type_management tempest.scenario.test_network_basic_ops|^test_security_groups_basic_ops|^test_server_basic_ops|^test_network_advanced_server_ops|^test_server_advanced_ops\" "	


detailed logs can be found at 
http://15.126.220.115/OVSVAPP-VLAN-KILO/602/203586/9215e7309301fd72961e906a1ae5afe1d5c8f909/ 

might have been deleted concurrently.
2015-07-21 12:27:55.595 WARNING neutron.db.agentschedulers_db [req-2585e7c4-8f33-439b-8b64-afc1522a686a None None] Removing network 17d6770a-c478-46d4-9556-ee77bbaf2d6b from agent 04fa72de-c1f9-4b51-8369-3b027aaf42d4 because the agent did not report to the server in the last 150 seconds.
2015-07-21 12:28:15.822 ERROR oslo_messaging.rpc.dispatcher [req-99ec4c7c-3f9d-4dd7-b920-c57606a07df7 None None] Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 281, in report_state
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     self.plugin.create_or_update_agent(context, agent_state)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 238, in create_or_update_agent
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return self._create_or_update_agent(context, agent)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 217, in _create_or_update_agent
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     context, agent['agent_type'], agent['host'])
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 193, in _get_agent_by_type_and_host
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     Agent.host == host).one()
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2395, in one
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     ret = list(self)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return self._execute_and_instances(context)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2451, in _execute_and_instances
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     close_with_result=True)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2442, in _connection_from_session
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     **kw)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 854, in connection
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     close_with_result=close_with_result)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 858, in _connection_for_bind
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return self.transaction._connection_for_bind(engine)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 322, in _connection_for_bind
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     conn = bind.contextual_connect()
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 208, in contextual_connect
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     self._wrap_pool_connect(self.pool.connect, None),
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 215, in _wrap_pool_connect
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return fn()
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 338, in connect
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return _ConnectionFairy._checkout(self)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 644, in _checkout
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     fairy = _ConnectionRecord.checkout(pool)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 440, in checkout
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     rec = pool._do_get()
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 959, in _do_get
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     (self.size(), self.overflow(), self._timeout))
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher 
2015-07-21 12:28:15.885 ERROR oslo_messaging.rpc.dispatcher [req-6ee592e3-e5aa-4d6d-a422-3e7ca4734a25 None None] Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10

** Affects: neutron (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1477077

Title:
  Exception during message handling: QueuePool limit of size 10 overflow
  20 reached, connection timed out, timeout 10 stable/kilo

Status in neutron package in Ubuntu:
  New

Bug description:
  OpenStack stable/kilo release, Ubuntu 14.04 devstack setup .while
  multiple calls are generated during tempest run(tempest.api.networks)
  in HP networking CI setup we see queuePool limit related exception
  which results in tempest failure.

  tempest tests that are run include
  cd /opt/stack/tempest; testr init ; ./tools/pretty_tox_serial.sh    \"  tempest.api.network.test_networks tempest.api.network.test_extensions tempest.api.network.test_allowed_address_pair tempest.api.network.test_extra_dhcp_options tempest.api.network.test_floating_ips_negative tempest.api.network.test_floating_ips tempest.api.network.test_networks_negative tempest.api.network.test_networks tempest.api.network.test_ports tempest.api.network.test_routers_negative tempest.api.network.test_routers tempest.api.network.test_security_groups_negative tempest.api.network.test_security_groups tempest.api.network.test_service_type_management tempest.scenario.test_network_basic_ops|^test_security_groups_basic_ops|^test_server_basic_ops|^test_network_advanced_server_ops|^test_server_advanced_ops\" "	


  detailed logs can be found at 
  http://15.126.220.115/OVSVAPP-VLAN-KILO/602/203586/9215e7309301fd72961e906a1ae5afe1d5c8f909/ 

  might have been deleted concurrently.
  2015-07-21 12:27:55.595 WARNING neutron.db.agentschedulers_db [req-2585e7c4-8f33-439b-8b64-afc1522a686a None None] Removing network 17d6770a-c478-46d4-9556-ee77bbaf2d6b from agent 04fa72de-c1f9-4b51-8369-3b027aaf42d4 because the agent did not report to the server in the last 150 seconds.
  2015-07-21 12:28:15.822 ERROR oslo_messaging.rpc.dispatcher [req-99ec4c7c-3f9d-4dd7-b920-c57606a07df7 None None] Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 281, in report_state
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     self.plugin.create_or_update_agent(context, agent_state)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 238, in create_or_update_agent
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return self._create_or_update_agent(context, agent)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 217, in _create_or_update_agent
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     context, agent['agent_type'], agent['host'])
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/db/agents_db.py", line 193, in _get_agent_by_type_and_host
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     Agent.host == host).one()
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2395, in one
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     ret = list(self)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return self._execute_and_instances(context)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2451, in _execute_and_instances
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     close_with_result=True)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2442, in _connection_from_session
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     **kw)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 854, in connection
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     close_with_result=close_with_result)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 858, in _connection_for_bind
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return self.transaction._connection_for_bind(engine)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 322, in _connection_for_bind
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     conn = bind.contextual_connect()
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 208, in contextual_connect
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     self._wrap_pool_connect(self.pool.connect, None),
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 215, in _wrap_pool_connect
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return fn()
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 338, in connect
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     return _ConnectionFairy._checkout(self)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 644, in _checkout
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     fairy = _ConnectionRecord.checkout(pool)
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 440, in checkout
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     rec = pool._do_get()
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 959, in _do_get
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher     (self.size(), self.overflow(), self._timeout))
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
  2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher 
  2015-07-21 12:28:15.885 ERROR oslo_messaging.rpc.dispatcher [req-6ee592e3-e5aa-4d6d-a422-3e7ca4734a25 None None] Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1477077/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list