[Bug 1812672] Re: impossible to create nova instances after upgrading to rocky

[Corey Bryant]

And available in the charm store.

** Changed in: charm-nova-cloud-controller
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1812672

Title:
  impossible to create nova instances after upgrading to rocky

Status in OpenStack nova-cloud-controller charm:
  Fix Released
Status in Ubuntu Cloud Archive:
  Fix Committed
Status in oslo.cache:
  Confirmed
Status in nova package in Ubuntu:
  Fix Released
Status in python-oslo.cache package in Ubuntu:
  Triaged

Bug description:
  Hi,

  I'm using bionic with 18.11 charms. I recently upgraded openstack from
  queens to rocky. After that, I was unable to create nova instances -
  they were stuck in BUILD state, without any error in nova-cloud-
  controller, neutron-api or keystone logs.

  While investigating, I noticed that "openstack compute service list"
  was empty, and this was generating an error in nova-api-os-compute.log
  (see below for the traceback).

  My investigations lead to
  https://github.com/openstack/oslo.cache/blob/master/oslo_cache/_memcache_pool.py#L48
  being the problem. If I comment out this line, then "openstack service
  list" works fine, and I can create instances without problem. I
  however don't know the consequences in the long run.

  Please advise.

  Thanks !

  
  nova-api-os-compute.log traceback when running "openstack service list" :

  2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 163, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self.queue.pop().connection
  2019-01-21 10:45:43.729 87283 ERROR nova.context IndexError: pop from an empty deque
  2019-01-21 10:45:43.729 87283 ERROR nova.context
  2019-01-21 10:45:43.729 87283 ERROR nova.context During handling of the above exception, another exception occurred:
  2019-01-21 10:45:43.729 87283 ERROR nova.context
  2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/context.py", line 438, in gather_result
  2019-01-21 10:45:43.729 87283 ERROR nova.context     result = fn(cctxt, *args, **kwargs)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2019-01-21 10:45:43.729 87283 ERROR nova.context     result = fn(cls, context, *args, **kwargs)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/objects/service.py", line 601, in get_all
  2019-01-21 10:45:43.729 87283 ERROR nova.context     context, db_services)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 88, in set_availability_zones
  2019-01-21 10:45:43.729 87283 ERROR nova.context     service['host'], az)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 108, in update_host_availability_zone_cache
  2019-01-21 10:45:43.729 87283 ERROR nova.context     cache.delete(cache_key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/cache_utils.py", line 122, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return self.region.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1002, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     self.backend.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/dogpile/cache/backends/memcached.py", line 188, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     self.client.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/backends/memcache_pool.py", line 31, in _run_method
  2019-01-21 10:45:43.729 87283 ERROR nova.context     with self.client_pool.acquire() as client:
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return next(self.gen)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 127, in acquire
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self.get(timeout=self._connection_get_timeout)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 295, in get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return self._get()
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 214, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = ConnectionPool._get(self)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 165, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self._create_connection()
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 206, in _create_connection
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return _MemcacheClient(self.urls, **self._arguments)
  2019-01-21 10:45:43.729 87283 ERROR nova.context TypeError: object() takes no parameters
  2019-01-21 10:45:43.729 87283 ERROR nova.context

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1812672/+subscriptions