[Bug 1822872] Re: Bionic: Luminous radosgw incompatible with libssl1.1

[Eric Desrochers]

It has been brought to my attention by an impacted user who tested a
test package [2.2.11-0ubuntu0.18.04.1+testpkg15042019b2] with the fix:

"
Hi,
..
All three machines are now running 12.2.11-0ubuntu0.18.04.1+testpkg15042019b2
[though installing those packages did restart the entire cluster, which isn't ideal]

...and yes, they are all listening on the https port, and some light testing with s3cmd suggests the S3 service is properly operational again.
..
"

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1822872

Title:
  Bionic: Luminous radosgw incompatible with libssl1.1

Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Bionic:
  In Progress

Bug description:
  [Impact]

  This is breaking Ceph cluster https service.

  # logs:
  2019-04-02 16:40:14.846313 7ff8c1736000 0 starting handler: civetweb
  2019-04-02 16:40:14.846397 7ff8c1736000 0 civetweb: 0x56114520d620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
  2019-04-02 16:40:14.846424 7ff8c1736000 -1 ERROR: failed run

  [Test Case]

  1) Generate a self-signed certificate or use whatever existing SSL
  certificate already in place.

  If one want to create a PEM file for civetweb, instructions can be found here :
  https://github.com/civetweb/civetweb/blob/master/docs/OpenSSL.md

  ** Note: "CivetWeb requires one certificate file in PEM format" **

  2) Enable logging and debugging in "/etc/ceph/ceph.conf"

  Example:
  ------
  log to syslog = true
  err to syslog = true
  clog to syslog = true
  debug rgw = 10/5
  debug civetweb = 1/10
  ------

  http://docs.ceph.com/docs/mimic/rados/troubleshooting/log-and-debug/

  3) From the radosgw node, modify "/etc/ceph/ceph.conf" as follow:
  rgw_frontends = civetweb port=443s ssl_certificate=/<path_to_PEM_FILE>/<PEM_FILE>

  4) Restart the daemon:
  systemctl restart ceph-radosgw at rgw.`hostname -s`

  5) Look logs:
  2019-04-10 12:02:53.535133 7fcd20c4e000 0 civetweb: 0x562d710ed620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks

  6) Look radosgw which should FAILED to start.
  systemctl status ceph-radosgw at rgw.`hostname -s`

  What we are looking for here is radosgw to be 'Active' and to have a
  LISTEN port on 443 as follow :

  $ netstat -anputa | grep LISTEN | grep 443 # or any port mentioned in the configuration above.
  tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      10153/radosgw

  [Potential Regression]

  * Same downgrade approach has been made for 'nodejs' via LP: #1798367

  * Nothing can be worst than current situation, considering that
  civetweb is non-functionnal when SSL is in used due to the
  incompatibility with 1.1 and make radosgw daemon to fail.

  * libssl1.0 and libssl1.1 are coinstallable ABIs so it shouldn't be a
  problem here.

  * See discussion IRC discussion (xnox/jamespage) on comment #11

  [Other Information]

  * Adding the OpenSSL 1.1 support has been explored and revealed to be non-trivial :
  https://github.com/civetweb/civetweb/pull/384/commits
  https://github.com/civetweb/civetweb/commit/adac9c916fa892ec5edce7b565803f1e62d304a2
  https://github.com/civetweb/civetweb/commit/5d83900fd29fb6fa1cd604676cb0562dc984dcc9

  http://docs.ceph.com/docs/bobtail/radosgw/troubleshooting/

  See discussion IRC discussion on comment #11

  [Original Description]

  Bionic's radosgw package (Version 12.2.11-0ubuntu0.18.04.1 ) can't run
  on Bionic, because the version of civetweb in Luminous is incompatible
  with libssl1.1, but it's built against libssl1.1.

  This has been known about upstream for a while now, and as noted in
  the bug-tracker (https://tracker.ceph.com/issues/20696), it can be
  fixed by building Luminous in an environment that has only libssl1.0
  available (or, in a more invasive manner, by incorporating a newer
  civetweb). A patch is in the tracker.ceph.com issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1822872/+subscriptions