[Bug 1822872] Re: Bionic: Luminous radosgw incompatible with libssl1.1
Eric Desrochers
eric.desrochers at canonical.com
Tue Apr 16 01:30:37 UTC 2019
** Description changed:
[Impact]
This is breaking Ceph cluster https service.
# logs:
2019-04-02 16:40:14.846313 7ff8c1736000 0 starting handler: civetweb
2019-04-02 16:40:14.846397 7ff8c1736000 0 civetweb: 0x56114520d620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
2019-04-02 16:40:14.846424 7ff8c1736000 -1 ERROR: failed run
[Test Case]
1) Generate a self-signed certificate or use whatever existing SSL
certificate already in place.
- 2) From the radosgw node, modify "/etc/ceph/ceph.conf" as follow:
- rgw_frontends = civetweb ssl_port=443 ssl_certificate=/etc/ssl/<CERTIFICATE>
+ If one want to create a PEM file for civetweb, instructions can be found here :
+ https://github.com/civetweb/civetweb/blob/master/docs/OpenSSL.md
+
+ ** Note: "CivetWeb requires one certificate file in PEM format" **
+
+ 2) Enable logging and debugging in "/etc/ceph/ceph.conf"
+
+ Example:
+ ------
+ log to syslog = true
+ err to syslog = true
+ clog to syslog = true
+ debug rgw = 10/5
+ debug civetweb = 1/10
+ ------
+
+ http://docs.ceph.com/docs/mimic/rados/troubleshooting/log-and-debug/
+
+ 3) From the radosgw node, modify "/etc/ceph/ceph.conf" as follow:
+ rgw_frontends = civetweb port=443s ssl_certificate=/<path_to_PEM_FILE>/<PEM_FILE>
3) Restart the daemon:
systemctl restart ceph-radosgw at rgw.`hostname -s`
4) Look logs:
2019-04-10 12:02:53.535133 7fcd20c4e000 0 civetweb: 0x562d710ed620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
5) Look radosgw which should FAILED to start.
+ systemctl status ceph-radosgw at rgw.`hostname -s`
+
+ What we are looking for here is radosgw to be 'Active' and to have a
+ LISTEN port on 443 as follow :
+
+ $ netstat -anputa | grep LISTEN | grep 443
+ tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 10153/radosgw
[Potential Regression]
* Same downgrade approach has been made for 'nodejs' via LP: #1798367
* Seems like civetweb is all that does SSL there, so it should be fine.
* Nothing can be worst than current situation, considering that civetweb
is non-functionnal when SSL is in used due to the incompatibility with
1.1 and make radosgw daemon to fail.
* libssl1.0 and libssl1.1 are coinstallable ABIs so it shouldn't be a
problem here.
* See discussion IRC discussion on comment #11
[Other Information]
* Adding the OpenSSL 1.1 support has been explored and revealed to be non-trivial :
https://github.com/civetweb/civetweb/pull/384/commits
https://github.com/civetweb/civetweb/commit/adac9c916fa892ec5edce7b565803f1e62d304a2
https://github.com/civetweb/civetweb/commit/5d83900fd29fb6fa1cd604676cb0562dc984dcc9
http://docs.ceph.com/docs/bobtail/radosgw/troubleshooting/
See discussion IRC discussion on comment #11
[Original Description]
Bionic's radosgw package (Version 12.2.11-0ubuntu0.18.04.1 ) can't run
on Bionic, because the version of civetweb in Luminous is incompatible
with libssl1.1, but it's built against libssl1.1.
This has been known about upstream for a while now, and as noted in the
bug-tracker (https://tracker.ceph.com/issues/20696), it can be fixed by
building Luminous in an environment that has only libssl1.0 available
(or, in a more invasive manner, by incorporating a newer civetweb). A
patch is in the tracker.ceph.com issue.
** Description changed:
[Impact]
This is breaking Ceph cluster https service.
# logs:
2019-04-02 16:40:14.846313 7ff8c1736000 0 starting handler: civetweb
2019-04-02 16:40:14.846397 7ff8c1736000 0 civetweb: 0x56114520d620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
2019-04-02 16:40:14.846424 7ff8c1736000 -1 ERROR: failed run
[Test Case]
1) Generate a self-signed certificate or use whatever existing SSL
certificate already in place.
If one want to create a PEM file for civetweb, instructions can be found here :
https://github.com/civetweb/civetweb/blob/master/docs/OpenSSL.md
** Note: "CivetWeb requires one certificate file in PEM format" **
2) Enable logging and debugging in "/etc/ceph/ceph.conf"
Example:
------
log to syslog = true
err to syslog = true
clog to syslog = true
debug rgw = 10/5
debug civetweb = 1/10
------
http://docs.ceph.com/docs/mimic/rados/troubleshooting/log-and-debug/
3) From the radosgw node, modify "/etc/ceph/ceph.conf" as follow:
rgw_frontends = civetweb port=443s ssl_certificate=/<path_to_PEM_FILE>/<PEM_FILE>
- 3) Restart the daemon:
+ 4) Restart the daemon:
systemctl restart ceph-radosgw at rgw.`hostname -s`
- 4) Look logs:
+ 5) Look logs:
2019-04-10 12:02:53.535133 7fcd20c4e000 0 civetweb: 0x562d710ed620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
- 5) Look radosgw which should FAILED to start.
+ 6) Look radosgw which should FAILED to start.
systemctl status ceph-radosgw at rgw.`hostname -s`
What we are looking for here is radosgw to be 'Active' and to have a
LISTEN port on 443 as follow :
$ netstat -anputa | grep LISTEN | grep 443
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 10153/radosgw
[Potential Regression]
* Same downgrade approach has been made for 'nodejs' via LP: #1798367
* Seems like civetweb is all that does SSL there, so it should be fine.
* Nothing can be worst than current situation, considering that civetweb
is non-functionnal when SSL is in used due to the incompatibility with
1.1 and make radosgw daemon to fail.
* libssl1.0 and libssl1.1 are coinstallable ABIs so it shouldn't be a
problem here.
* See discussion IRC discussion on comment #11
[Other Information]
* Adding the OpenSSL 1.1 support has been explored and revealed to be non-trivial :
https://github.com/civetweb/civetweb/pull/384/commits
https://github.com/civetweb/civetweb/commit/adac9c916fa892ec5edce7b565803f1e62d304a2
https://github.com/civetweb/civetweb/commit/5d83900fd29fb6fa1cd604676cb0562dc984dcc9
http://docs.ceph.com/docs/bobtail/radosgw/troubleshooting/
See discussion IRC discussion on comment #11
[Original Description]
Bionic's radosgw package (Version 12.2.11-0ubuntu0.18.04.1 ) can't run
on Bionic, because the version of civetweb in Luminous is incompatible
with libssl1.1, but it's built against libssl1.1.
This has been known about upstream for a while now, and as noted in the
bug-tracker (https://tracker.ceph.com/issues/20696), it can be fixed by
building Luminous in an environment that has only libssl1.0 available
(or, in a more invasive manner, by incorporating a newer civetweb). A
patch is in the tracker.ceph.com issue.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1822872
Title:
Bionic: Luminous radosgw incompatible with libssl1.1
Status in ceph package in Ubuntu:
Fix Released
Status in ceph source package in Bionic:
In Progress
Bug description:
[Impact]
This is breaking Ceph cluster https service.
# logs:
2019-04-02 16:40:14.846313 7ff8c1736000 0 starting handler: civetweb
2019-04-02 16:40:14.846397 7ff8c1736000 0 civetweb: 0x56114520d620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
2019-04-02 16:40:14.846424 7ff8c1736000 -1 ERROR: failed run
[Test Case]
1) Generate a self-signed certificate or use whatever existing SSL
certificate already in place.
If one want to create a PEM file for civetweb, instructions can be found here :
https://github.com/civetweb/civetweb/blob/master/docs/OpenSSL.md
** Note: "CivetWeb requires one certificate file in PEM format" **
2) Enable logging and debugging in "/etc/ceph/ceph.conf"
Example:
------
log to syslog = true
err to syslog = true
clog to syslog = true
debug rgw = 10/5
debug civetweb = 1/10
------
http://docs.ceph.com/docs/mimic/rados/troubleshooting/log-and-debug/
3) From the radosgw node, modify "/etc/ceph/ceph.conf" as follow:
rgw_frontends = civetweb port=443s ssl_certificate=/<path_to_PEM_FILE>/<PEM_FILE>
4) Restart the daemon:
systemctl restart ceph-radosgw at rgw.`hostname -s`
5) Look logs:
2019-04-10 12:02:53.535133 7fcd20c4e000 0 civetweb: 0x562d710ed620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
6) Look radosgw which should FAILED to start.
systemctl status ceph-radosgw at rgw.`hostname -s`
What we are looking for here is radosgw to be 'Active' and to have a
LISTEN port on 443 as follow :
$ netstat -anputa | grep LISTEN | grep 443
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 10153/radosgw
[Potential Regression]
* Same downgrade approach has been made for 'nodejs' via LP: #1798367
* Seems like civetweb is all that does SSL there, so it should be
fine.
* Nothing can be worst than current situation, considering that
civetweb is non-functionnal when SSL is in used due to the
incompatibility with 1.1 and make radosgw daemon to fail.
* libssl1.0 and libssl1.1 are coinstallable ABIs so it shouldn't be a
problem here.
* See discussion IRC discussion on comment #11
[Other Information]
* Adding the OpenSSL 1.1 support has been explored and revealed to be non-trivial :
https://github.com/civetweb/civetweb/pull/384/commits
https://github.com/civetweb/civetweb/commit/adac9c916fa892ec5edce7b565803f1e62d304a2
https://github.com/civetweb/civetweb/commit/5d83900fd29fb6fa1cd604676cb0562dc984dcc9
http://docs.ceph.com/docs/bobtail/radosgw/troubleshooting/
See discussion IRC discussion on comment #11
[Original Description]
Bionic's radosgw package (Version 12.2.11-0ubuntu0.18.04.1 ) can't run
on Bionic, because the version of civetweb in Luminous is incompatible
with libssl1.1, but it's built against libssl1.1.
This has been known about upstream for a while now, and as noted in
the bug-tracker (https://tracker.ceph.com/issues/20696), it can be
fixed by building Luminous in an environment that has only libssl1.0
available (or, in a more invasive manner, by incorporating a newer
civetweb). A patch is in the tracker.ceph.com issue.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1822872/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list