[Bug 1812672] Fix merged to charm-nova-cloud-controller (stable/18.11)

Thu Feb 21 09:16:25 UTC 2019

Reviewed:  https://review.openstack.org/637149
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=9092ab63678962bb942b7e0ff03438c32147d60a
Submitter: Zuul
Branch:    stable/18.11

commit 9092ab63678962bb942b7e0ff03438c32147d60a
Author: Sahid Orentino Ferdjaoui <sahid.ferdjaoui at canonical.com>
Date:   Mon Jan 28 11:37:40 2019 +0100

    service: updates nova-api-os-compute service to use apache wsgi

    Due to an issue in python3 oslo_cache+eventlet when using
    memcached. As workaroud for Rocky it has been decided to run service
    nova-api-os-compute from systemd to apache2.

    Conflicts:
      hooks/nova_cc_hooks.py
        in upgrade_charm() the conflict were because i did not
        cherrypicked the template update for placement api
        (131497868f442bcf06bc199a36e2962b8ac0018d) which does not make
        sens for the purpose of a backport to stable. I removed the
        comment.

      hooks/nova_cc_utils.py
        in disable_package_apache_site() the conflict where also related
        to (131497868f442bcf06bc199a36e2962b8ac0018d).

    Closes-Bug: #1812672
    Change-Id: I3bf279638c5decf1020345f3d2e876e379144997
    Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui at canonical.com>
    (cherry picked from commit 13eca55803c8320a401ccf923c29b7bfeb85d0fc)

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1812672

Title:
  impossible to create nova instances after upgrading to rocky

Status in OpenStack nova-cloud-controller charm:
  Fix Committed
Status in Ubuntu Cloud Archive:
  Fix Committed
Status in oslo.cache:
  Confirmed
Status in nova package in Ubuntu:
  Fix Released
Status in python-oslo.cache package in Ubuntu:
  Triaged

Bug description:
  Hi,

  I'm using bionic with 18.11 charms. I recently upgraded openstack from
  queens to rocky. After that, I was unable to create nova instances -
  they were stuck in BUILD state, without any error in nova-cloud-
  controller, neutron-api or keystone logs.

  While investigating, I noticed that "openstack compute service list"
  was empty, and this was generating an error in nova-api-os-compute.log
  (see below for the traceback).

  My investigations lead to
  https://github.com/openstack/oslo.cache/blob/master/oslo_cache/_memcache_pool.py#L48
  being the problem. If I comment out this line, then "openstack service
  list" works fine, and I can create instances without problem. I
  however don't know the consequences in the long run.

  Please advise.

  Thanks !

  nova-api-os-compute.log traceback when running "openstack service list" :

  2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 163, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self.queue.pop().connection
  2019-01-21 10:45:43.729 87283 ERROR nova.context IndexError: pop from an empty deque
  2019-01-21 10:45:43.729 87283 ERROR nova.context
  2019-01-21 10:45:43.729 87283 ERROR nova.context During handling of the above exception, another exception occurred:
  2019-01-21 10:45:43.729 87283 ERROR nova.context
  2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/context.py", line 438, in gather_result
  2019-01-21 10:45:43.729 87283 ERROR nova.context     result = fn(cctxt, *args, **kwargs)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2019-01-21 10:45:43.729 87283 ERROR nova.context     result = fn(cls, context, *args, **kwargs)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/objects/service.py", line 601, in get_all
  2019-01-21 10:45:43.729 87283 ERROR nova.context     context, db_services)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 88, in set_availability_zones
  2019-01-21 10:45:43.729 87283 ERROR nova.context     service['host'], az)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 108, in update_host_availability_zone_cache
  2019-01-21 10:45:43.729 87283 ERROR nova.context     cache.delete(cache_key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/cache_utils.py", line 122, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return self.region.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1002, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     self.backend.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/dogpile/cache/backends/memcached.py", line 188, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     self.client.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/backends/memcache_pool.py", line 31, in _run_method
  2019-01-21 10:45:43.729 87283 ERROR nova.context     with self.client_pool.acquire() as client:
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return next(self.gen)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 127, in acquire
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self.get(timeout=self._connection_get_timeout)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 295, in get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return self._get()
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 214, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = ConnectionPool._get(self)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 165, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self._create_connection()
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 206, in _create_connection
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return _MemcacheClient(self.urls, **self._arguments)
  2019-01-21 10:45:43.729 87283 ERROR nova.context TypeError: object() takes no parameters
  2019-01-21 10:45:43.729 87283 ERROR nova.context

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1812672/+subscriptions