[Bug 1817055] Re: Broken dependency when setting swift-proxy on Bionic

Mathew Hodson mathew.hodson at gmail.com
Sun Mar 10 08:05:59 UTC 2019 

** Changed in: python-eventlet (Ubuntu Disco)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-eventlet in Ubuntu.
https://bugs.launchpad.net/bugs/1817055

Title:
  Broken dependency when setting swift-proxy on Bionic

Status in OpenStack swift-proxy charm:
  Invalid
Status in python-eventlet package in Ubuntu:
  Fix Released
Status in python-eventlet source package in Bionic:
  Fix Committed
Status in python-eventlet source package in Cosmic:
  Fix Committed
Status in python-eventlet source package in Disco:
  Fix Released

Bug description:
  [Impact] 
  Swift proxy server will sometimes fail to start due to racy-ness in monkey patching by eventlet and daemon startup.  Swift dies, no object storage.

  [Test Case]
  Deploy swift using the reference bundles for deploying OpenStack with Juju and the OpenStack charms.
  Depending on the number of cores being used the stacktrace in the original bug report will be seen and swift-proxy will fail to startup.

  [Regression Potential] 
  The workaround fix to ensure that the eventlet hub is correctly initialised has been in the upstream codebase since 2017 so low potential for regression.

  [Original Bug Report]
  Charm options:
    swift-proxy:
      charm: cs:swift-proxy
      num_units: 3
      bindings:
        "": *oam-space
        public: *public-space
        admin: *admin-space
        internal: *internal-space
        swift-storage: *swift-cluster-space
      options:
        openstack-origin: *openstack-origin
        vip: *swift-proxy-vip
        region: *openstack-region
        zone-assignment: manual
        partition-power: *partition-power
        workers: *workers
      to:
      - lxd:0
      - lxd:1
      - lxd:2

  Setting up swift-proxy on Bionic causes swift-proxy.service to fail at
  start with:

  Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: Stopped LSB: Swift proxy server.
  Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: swift-proxy.service: Failed to reset devices.list: Operation not permitted
  Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: Starting LSB: Swift proxy server...
  Feb 21 11:16:07 juju-29125a-0-lxd-15 swift-proxy[41939]:  * Starting Swift Swift proxy server swift-init proxy-server
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: Starting proxy-server...(/etc/swift/proxy-server.conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: Traceback (most recent call last):
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/bin/swift-proxy-server", line 23, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 912, in run_wsgi
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     loadapp(conf_path, global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 390, in loadapp
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 374, in loadcontext
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     return loader.get_context(object_type, name, global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 66, in get_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     object_type, name=name, global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     global_additions=global_additions)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     for name in pipeline[:-1]]
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 66, in get_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     object_type, name=name, global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 458, in get_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     section)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 517, in _context_from_explicit
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     value = import_string(found_expr)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 22, in import_string
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     return pkg_resources.EntryPoint.parse("x=" + s).load(False)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2324, in load
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     return self.resolve()
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     module = __import__(self.module_name, fromlist=['__name__'], level=0)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 221, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from keystoneauth1 import adapter
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 16, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from keystoneauth1 import session
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 40, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     import osprofiler.web as osprofiler_web
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 20, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from osprofiler import profiler
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 26, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from osprofiler import notifier
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/notifier.py", line 16, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from osprofiler.drivers import base
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/drivers/__init__.py", line 4, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from osprofiler.drivers import loginsight  # noqa
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/drivers/loginsight.py", line 24, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from oslo_concurrency.lockutils import synchronized
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 27, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     import fasteners
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/fasteners/__init__.py", line 23, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from fasteners.lock import locked  # noqa
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/fasteners/lock.py", line 24, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from fasteners import _utils
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/fasteners/_utils.py", line 22, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from monotonic import monotonic as now  # noqa
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/monotonic.py", line 164, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     raise RuntimeError('no suitable implementation for this system')
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: RuntimeError: no suitable implementation for this system
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:    ...fail!
  Feb 21 11:16:08 juju-29125a-0-lxd-15 systemd[1]: Started LSB: Swift proxy server.

  Looking at errors for monotonic lib, I can find other references around github:
  https://github.com/nameko/nameko/issues/533
  https://github.com/eventlet/eventlet/issues/401

  Swift-proxy indeed depends on eventlet:
  root at juju-29125a-0-lxd-15:~# dpkg -l | grep eventlet
  ii  python-eventlet                0.20.0-4                                     all          concurrent networking library - Python 2.x

  And then the issue described at:
  https://github.com/eventlet/eventlet/issues/401 applies.

  Moving back to Xenial seems to stabilize swift proxy.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-swift-proxy/+bug/1817055/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list