[Bug 1812672] Re: impossible to create nova instances after upgrading to rocky

Wed Apr 24 14:15:03 UTC 2019

Reviewed:  https://review.opendev.org/640500
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=caf5443de8ff4ff353d741a125f782c67f579f52
Submitter: Zuul
Branch:    stable/rocky

commit caf5443de8ff4ff353d741a125f782c67f579f52
Author: Ben Nemec <bnemec at redhat.com>
Date:   Tue Feb 26 22:12:23 2019 +0000

    Fix memcache pool client in monkey-patched environments

    First off, this is an ugly hack, but we're dealing with code that
    essentially monkey-patches a monkey-patch. You reap what you sow.

    Per the linked bug, our connection pool client explodes on python 3
    with eventlet monkey-patching in force:

    TypeError: object() takes no parameters

    This is due to the way __new__ is overridden in the class. We need
    to strip arguments from the call before they get to object(), which
    doesn't accept args.

    Unfortunately, when we're _not_ monkey-patched, adding the new
    override implementation fails with:

    TypeError: object.__new__(_MemcacheClient) is not safe,
    use Client.__new__()

    As such, we need different implementations depending on whether we
    are monkey-patched or not. This change passes both with and without
    monkey-patching and adds a unit test that exposes the bug.

    Note that this is a temporary, backportable fix that will ultimately
    be replaced by a switch to the pymemcache library which does not
    have the threading.local problem being worked around here.

    Change-Id: I039dffadeebd0ff4479b9c870c257772c43aba53
    Partial-Bug: 1812935
    Closes-Bug: 1812672
    (cherry picked from commit f4a25f642991a7114b86f6eb7d0bac3d599953a6)

** Tags added: in-stable-rocky

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1812672

Title:
  impossible to create nova instances after upgrading to rocky

Status in OpenStack nova-cloud-controller charm:
  Fix Released
Status in Ubuntu Cloud Archive:
  Fix Released
Status in oslo.cache:
  Confirmed
Status in nova package in Ubuntu:
  Fix Released
Status in python-oslo.cache package in Ubuntu:
  Triaged

Bug description:
  Hi,

  I'm using bionic with 18.11 charms. I recently upgraded openstack from
  queens to rocky. After that, I was unable to create nova instances -
  they were stuck in BUILD state, without any error in nova-cloud-
  controller, neutron-api or keystone logs.

  While investigating, I noticed that "openstack compute service list"
  was empty, and this was generating an error in nova-api-os-compute.log
  (see below for the traceback).

  My investigations lead to
  https://github.com/openstack/oslo.cache/blob/master/oslo_cache/_memcache_pool.py#L48
  being the problem. If I comment out this line, then "openstack service
  list" works fine, and I can create instances without problem. I
  however don't know the consequences in the long run.

  Please advise.

  Thanks !

  nova-api-os-compute.log traceback when running "openstack service list" :

  2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 163, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self.queue.pop().connection
  2019-01-21 10:45:43.729 87283 ERROR nova.context IndexError: pop from an empty deque
  2019-01-21 10:45:43.729 87283 ERROR nova.context
  2019-01-21 10:45:43.729 87283 ERROR nova.context During handling of the above exception, another exception occurred:
  2019-01-21 10:45:43.729 87283 ERROR nova.context
  2019-01-21 10:45:43.729 87283 ERROR nova.context Traceback (most recent call last):
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/context.py", line 438, in gather_result
  2019-01-21 10:45:43.729 87283 ERROR nova.context     result = fn(cctxt, *args, **kwargs)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2019-01-21 10:45:43.729 87283 ERROR nova.context     result = fn(cls, context, *args, **kwargs)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/objects/service.py", line 601, in get_all
  2019-01-21 10:45:43.729 87283 ERROR nova.context     context, db_services)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 88, in set_availability_zones
  2019-01-21 10:45:43.729 87283 ERROR nova.context     service['host'], az)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/availability_zones.py", line 108, in update_host_availability_zone_cache
  2019-01-21 10:45:43.729 87283 ERROR nova.context     cache.delete(cache_key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/nova/cache_utils.py", line 122, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return self.region.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1002, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     self.backend.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/dogpile/cache/backends/memcached.py", line 188, in delete
  2019-01-21 10:45:43.729 87283 ERROR nova.context     self.client.delete(key)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/backends/memcache_pool.py", line 31, in _run_method
  2019-01-21 10:45:43.729 87283 ERROR nova.context     with self.client_pool.acquire() as client:
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return next(self.gen)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 127, in acquire
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self.get(timeout=self._connection_get_timeout)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 295, in get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return self._get()
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 214, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = ConnectionPool._get(self)
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 165, in _get
  2019-01-21 10:45:43.729 87283 ERROR nova.context     conn = self._create_connection()
  2019-01-21 10:45:43.729 87283 ERROR nova.context   File "/usr/lib/python3/dist-packages/oslo_cache/_memcache_pool.py", line 206, in _create_connection
  2019-01-21 10:45:43.729 87283 ERROR nova.context     return _MemcacheClient(self.urls, **self._arguments)
  2019-01-21 10:45:43.729 87283 ERROR nova.context TypeError: object() takes no parameters
  2019-01-21 10:45:43.729 87283 ERROR nova.context

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1812672/+subscriptions