[Bug 1817055] Re: Broken dependency when setting swift-proxy on Bionic
Pedro GuimarĂ£es
1817055 at bugs.launchpad.net
Sun Feb 24 10:50:04 UTC 2019
Well, I've found out why charm is not complaining: systemctl is marking
as SUCCESS despite failures on journalctl.
Also, running an strace on this, I see:
close(3) = 0
stat("/etc/swift/proxy-server.conf", {st_mode=S_IFREG|0644, st_size=2797, ...}) = 0
openat(AT_FDCWD, "/var/run/swift", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
write(1, "Starting proxy-server...(/etc/sw"..., 55Starting proxy-server...(/etc/swift/proxy-server.conf)
) = 55
pipe([3, 4]) = 0
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
fcntl(4, F_GETFD) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
pipe([5, 6]) = 0
fcntl(5, F_GETFD) = 0
fcntl(5, F_SETFD, FD_CLOEXEC) = 0
fcntl(6, F_GETFD) = 0
fcntl(6, F_SETFD, FD_CLOEXEC) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f3707c16a10) = -1 EAGAIN (Resource temporarily unavailable)
So, my guess is that control nodes are overloaded with containers and swift cannot spawn any more workers.
I moved swift-proxy to swift-storage nodes and it worked.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-eventlet in Ubuntu.
https://bugs.launchpad.net/bugs/1817055
Title:
Broken dependency when setting swift-proxy on Bionic
Status in OpenStack swift-proxy charm:
Invalid
Status in python-eventlet package in Ubuntu:
Fix Released
Status in python-eventlet source package in Bionic:
In Progress
Status in python-eventlet source package in Cosmic:
In Progress
Status in python-eventlet source package in Disco:
Fix Released
Bug description:
Charm options:
swift-proxy:
charm: cs:swift-proxy
num_units: 3
bindings:
"": *oam-space
public: *public-space
admin: *admin-space
internal: *internal-space
swift-storage: *swift-cluster-space
options:
openstack-origin: *openstack-origin
vip: *swift-proxy-vip
region: *openstack-region
zone-assignment: manual
partition-power: *partition-power
workers: *workers
to:
- lxd:0
- lxd:1
- lxd:2
Setting up swift-proxy on Bionic causes swift-proxy.service to fail at start with:
Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: Stopped LSB: Swift proxy server.
Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: swift-proxy.service: Failed to reset devices.list: Operation not permitted
Feb 21 11:16:07 juju-29125a-0-lxd-15 systemd[1]: Starting LSB: Swift proxy server...
Feb 21 11:16:07 juju-29125a-0-lxd-15 swift-proxy[41939]: * Starting Swift Swift proxy server swift-init proxy-server
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: Starting proxy-server...(/etc/swift/proxy-server.conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: Traceback (most recent call last):
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/bin/swift-proxy-server", line 23, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 912, in run_wsgi
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: loadapp(conf_path, global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 390, in loadapp
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 374, in loadcontext
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: return loader.get_context(object_type, name, global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 66, in get_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: object_type, name=name, global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: global_additions=global_additions)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: for name in pipeline[:-1]]
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 66, in get_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: object_type, name=name, global_conf=global_conf)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 458, in get_context
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: section)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 517, in _context_from_explicit
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: value = import_string(found_expr)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 22, in import_string
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2324, in load
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: return self.resolve()
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: module = __import__(self.module_name, fromlist=['__name__'], level=0)
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 221, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from keystoneauth1 import adapter
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 16, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from keystoneauth1 import session
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 40, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: import osprofiler.web as osprofiler_web
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/web.py", line 20, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from osprofiler import profiler
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 26, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from osprofiler import notifier
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/notifier.py", line 16, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from osprofiler.drivers import base
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/drivers/__init__.py", line 4, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from osprofiler.drivers import loginsight # noqa
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/osprofiler/drivers/loginsight.py", line 24, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from oslo_concurrency.lockutils import synchronized
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 27, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: import fasteners
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/fasteners/__init__.py", line 23, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from fasteners.lock import locked # noqa
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/fasteners/lock.py", line 24, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from fasteners import _utils
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/fasteners/_utils.py", line 22, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: from monotonic import monotonic as now # noqa
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: File "/usr/lib/python2.7/dist-packages/monotonic.py", line 164, in <module>
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: raise RuntimeError('no suitable implementation for this system')
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: RuntimeError: no suitable implementation for this system
Feb 21 11:16:08 juju-29125a-0-lxd-15 swift-proxy[41939]: ...fail!
Feb 21 11:16:08 juju-29125a-0-lxd-15 systemd[1]: Started LSB: Swift proxy server.
Looking at errors for monotonic lib, I can find other references around github:
https://github.com/nameko/nameko/issues/533
https://github.com/eventlet/eventlet/issues/401
Swift-proxy indeed depends on eventlet:
root at juju-29125a-0-lxd-15:~# dpkg -l | grep eventlet
ii python-eventlet 0.20.0-4 all concurrent networking library - Python 2.x
And then the issue described at:
https://github.com/eventlet/eventlet/issues/401 applies.
Moving back to Xenial seems to stabilize swift proxy.
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-swift-proxy/+bug/1817055/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list