[Bug 1812672] Re: impossible to create nova instances after upgrading to rocky
Sahid Orentino
sahid.ferdjaoui at canonical.com
Tue Jan 22 11:00:18 UTC 2019
I understand your point regarding Python versions but still I don't see
any incompatible changes and commenting that line should not make
differences.
I noticed something strange in logs that you shared to use. The memcache
module used is coming from a package called "dogpile" /usr/lib/python3
/dist-packages/dogpile where it should normally come from
/usr/lib/python3/dist-packages/memcache.py
Can you try to log in a Python prompt, import "memcache" and then print
the version and path?
My thinking is that there is a package conflict.
You should have something like:
(.py3) ubuntu at bug1812672:~$ python
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import memcache
>>> memcache.__version__
'1.57'
>>> memcache.__file__
'/usr/lib/python3/dist-packages/memcache.py'
(.py2) ubuntu at bug1812672:~$ python
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import memcache
>>> memcache.__version__
'1.57'
>>> memcache.__file__
'/usr/lib/python2.7/dist-packages/memcache.pyc'
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1812672
Title:
impossible to create nova instances after upgrading to rocky
Status in Ubuntu Cloud Archive:
New
Bug description:
Hi,
I'm using bionic with 18.11 charms. I recently upgraded openstack from
queens to rocky. After that, I was unable to create nova instances -
they were stuck in BUILD state, without any error in nova-cloud-
controller, neutron-api or keystone logs.
While investigating, I noticed that "openstack compute service list"
was empty, and this was generating an error in nova-api-os-compute.log
(see below for the traceback).
My investigations lead to
https://github.com/openstack/oslo.cache/blob/master/oslo_cache/_memcache_pool.py#L48
being the problem. If I comment out this line, then "openstack service
list" works fine, and I can create instances without problem. I
however don't know the consequences in the long run.
Please advise.
Thanks !
nova-api-os-compute.log traceback when running "openstack service list" :
2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 163, in _get
2019-01-21 10:45:43.729 87283 ERROR nova.context conn = self.queue.pop().connection
2019-01-21 10:45:43.729 87283 ERROR nova.context IndexError: pop from an empty deque
2019-01-21 10:45:43.729 87283 ERROR nova.context
2019-01-21 10:45:43.729 87283 ERROR nova.context During handling of the above exception, another exception occurred:
2019-01-21 10:45:43.729 87283 ERROR nova.context
2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/nova/context.py", line 438, in gather_result
2019-01-21 10:45:43.729 87283 ERROR nova.context result = fn(cctxt, *args, **kwargs)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-01-21 10:45:43.729 87283 ERROR nova.context result = fn(cls, context, *args, **kwargs)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/nova/objects/service.py", line 601, in get_all
2019-01-21 10:45:43.729 87283 ERROR nova.context context, db_services)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 88, in set_availability_zones
2019-01-21 10:45:43.729 87283 ERROR nova.context service['host'], az)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 108, in update_host_availability_zone_cache
2019-01-21 10:45:43.729 87283 ERROR nova.context cache.delete(cache_key)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/nova/cache_utils.py", line 122, in delete
2019-01-21 10:45:43.729 87283 ERROR nova.context return self.region.delete(key)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1002, in delete
2019-01-21 10:45:43.729 87283 ERROR nova.context self.backend.delete(key)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/dogpile/cache/backends/memcached.py", line 188, in delete
2019-01-21 10:45:43.729 87283 ERROR nova.context self.client.delete(key)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/oslo_cache/backends/memcache_pool.py", line 31, in _run_method
2019-01-21 10:45:43.729 87283 ERROR nova.context with self.client_pool.acquire() as client:
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
2019-01-21 10:45:43.729 87283 ERROR nova.context return next(self.gen)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 127, in acquire
2019-01-21 10:45:43.729 87283 ERROR nova.context conn = self.get(timeout=self._connection_get_timeout)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 295, in get
2019-01-21 10:45:43.729 87283 ERROR nova.context return self._get()
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 214, in _get
2019-01-21 10:45:43.729 87283 ERROR nova.context conn = ConnectionPool._get(self)
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 165, in _get
2019-01-21 10:45:43.729 87283 ERROR nova.context conn = self._create_connection()
2019-01-21 10:45:43.729 87283 ERROR nova.context File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 206, in _create_connection
2019-01-21 10:45:43.729 87283 ERROR nova.context return _MemcacheClient(self.urls, **self._arguments)
2019-01-21 10:45:43.729 87283 ERROR nova.context TypeError: object() takes no parameters
2019-01-21 10:45:43.729 87283 ERROR nova.context
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1812672/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list