[Bug 1817055] Re: Broken dependency when setting swift-proxy on Bionic

Pedro Guimarães 1817055 at bugs.launchpad.net
Sat Feb 23 12:15:20 UTC 2019 

The problem disappears when switching to your PPA. However, I still see
some trace on journalctl for swift-proxy:
https://pastebin.ubuntu.com/p/RFzk7qRQ6J/

The process marks: 
● swift-proxy.service - LSB: Swift proxy server
   Loaded: loaded (/etc/init.d/swift-proxy; generated)
   Active: active (exited) since Sat 2019-02-23 12:03:14 UTC; 1min 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 54857 ExecStop=/etc/init.d/swift-proxy stop (code=exited, status=0/SUCCESS)
  Process: 54865 ExecStart=/etc/init.d/swift-proxy start (code=exited, status=0/SUCCESS)

Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]:     pid = self.spawn(conf_file, **kwargs)
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]:   File "/usr/lib/python2.7/dist-packages/swift/common/manager.py", line 669, in spawn
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]:     proc = subprocess.Popen(args, stdout=re_out, stderr=re_err)
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]:   File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]:     errread, errwrite)
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]:   File "/usr/lib/python2.7/subprocess.py", line 938, in _execute_child
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]:     self.pid = os.fork()
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]: OSError: [Errno 11] Resource temporarily unavailable
Feb 23 12:03:14 juju-b57129-2-lxd-17 swift-proxy[54865]:    ...fail!
Feb 23 12:03:14 juju-b57129-2-lxd-17 systemd[1]: Started LSB: Swift proxy server.


Although it marks as Started, I fail to connect to swift with: https://pastebin.canonical.com/p/m23rCwsTjS/

The VIP is correctly allocated to one swift-proxy instance.

I dont see any process referring to swift-proxy on my lxd:
root       823  0.0  0.0  21768  3524 ?        Ss   11:36   0:00 bash /lib/systemd/system/jujud-unit-swift-proxy-3/exec-start.sh
root       827  0.0  0.0 3617772 75964 ?       Sl   11:36   0:07 /var/lib/juju/tools/unit-swift-proxy-3/jujud unit --data-dir /var/lib/juju --unit-name swift-proxy/3 --debug
root     16347  0.0  0.0  21768  3512 ?        Ss   11:39   0:00 bash /lib/systemd/system/jujud-unit-hacluster-swift-5/exec-start.sh
root     16362  0.0  0.0 3543528 70764 ?       Sl   11:39   0:02 /var/lib/juju/tools/unit-hacluster-swift-5/jujud unit --data-dir /var/lib/juju --unit-name hacluster-swift/5 --debug
ubuntu   70915  0.0  0.0  13160  1076 pts/0    S+   12:14   0:00 grep --color=auto swift

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-eventlet in Ubuntu.
https://bugs.launchpad.net/bugs/1817055

Title:
  Broken dependency when setting swift-proxy on Bionic

Status in OpenStack swift-proxy charm:
  Invalid
Status in python-eventlet package in Ubuntu:
  Fix Released
Status in python-eventlet source package in Bionic:
  In Progress
Status in python-eventlet source package in Cosmic:
  In Progress
Status in python-eventlet source package in Disco:
  Fix Released

Bug description:
  Charm options:
    swift-proxy:
      charm: cs:swift-proxy
      num_units: 3
      bindings:
        "": *oam-space
        public: *public-space
        admin: *admin-space
        internal: *internal-space
        swift-storage: *swift-cluster-space
      options:
        openstack-origin: *openstack-origin
        vip: *swift-proxy-vip
        region: *openstack-region
        zone-assignment: manual
        partition-power: *partition-power
        workers: *workers
      to:
      - lxd:0
      - lxd:1
      - lxd:2

  
  Setting up swift-proxy on Bionic causes swift-proxy.service to fail at start with:

  Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: Stopped LSB: Swift proxy server.
  Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: swift-proxy.service: Failed to reset devices.list: Operation not permitted
  Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: Starting LSB: Swift proxy server...
  Feb 21 11:16:07 juju-29125a-0-lxd-15 swift-proxy[41939]:  * Starting Swift Swift proxy server swift-init proxy-server
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: Starting proxy-server...(/etc/swift/proxy-server.conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: Traceback (most recent call last):
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/bin/swift-proxy-server", line 23, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 912, in run_wsgi
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     loadapp(conf_path, global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 390, in loadapp
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 374, in loadcontext
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     return loader.get_context(object_type, name, global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 66, in get_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     object_type, name=name, global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     global_additions=global_additions)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     for name in pipeline[:-1]]
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 66, in get_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     object_type, name=name, global_conf=global_conf)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 458, in get_context
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     section)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 517, in _context_from_explicit
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     value = import_string(found_expr)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 22, in import_string
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     return pkg_resources.EntryPoint.parse("x=" + s).load(False)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2324, in load
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     return self.resolve()
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     module = __import__(self.module_name, fromlist=['__name__'], level=0)
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 221, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from keystoneauth1 import adapter
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 16, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from keystoneauth1 import session
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 40, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     import osprofiler.web as osprofiler_web
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 20, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from osprofiler import profiler
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 26, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from osprofiler import notifier
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/notifier.py", line 16, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from osprofiler.drivers import base
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/drivers/__init__.py", line 4, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from osprofiler.drivers import loginsight  # noqa
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/osprofiler/drivers/loginsight.py", line 24, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from oslo_concurrency.lockutils import synchronized
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 27, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     import fasteners
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/fasteners/__init__.py", line 23, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from fasteners.lock import locked  # noqa
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/fasteners/lock.py", line 24, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from fasteners import _utils
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/fasteners/_utils.py", line 22, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     from monotonic import monotonic as now  # noqa
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:   File "/usr/lib/python2.7/dist-packages/monotonic.py", line 164, in <module>
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:     raise RuntimeError('no suitable implementation for this system')
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: RuntimeError: no suitable implementation for this system
  Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]:    ...fail!
  Feb 21 11:16:08 juju-29125a-0-lxd-15 systemd[1]: Started LSB: Swift proxy server.


  Looking at errors for monotonic lib, I can find other references around github:
  https://github.com/nameko/nameko/issues/533
  https://github.com/eventlet/eventlet/issues/401

  Swift-proxy indeed depends on eventlet:
  root at juju-29125a-0-lxd-15:~# dpkg -l | grep eventlet
  ii  python-eventlet                0.20.0-4                                     all          concurrent networking library - Python 2.x

  And then the issue described at:
  https://github.com/eventlet/eventlet/issues/401 applies.

  Moving back to Xenial seems to stabilize swift proxy.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-swift-proxy/+bug/1817055/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list