[Bug 1822872] Re: Bionic: Luminous radosgw incompatible with libssl1.1

[Eric Desrochers]

The backport of the civetweb fix in the attempts to resolve the
compatibility problem, reveal not being not being trivial, as it was
made on top of things not yet present in v1.8.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1822872

Title:
  Bionic: Luminous radosgw incompatible with libssl1.1

Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Bionic:
  In Progress

Bug description:
  [Impact]

  This is breaking Ceph cluster https service.

  # logs:
  2019-04-02 16:40:14.846313 7ff8c1736000 0 starting handler: civetweb
  2019-04-02 16:40:14.846397 7ff8c1736000 0 civetweb: 0x56114520d620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks
  2019-04-02 16:40:14.846424 7ff8c1736000 -1 ERROR: failed run

  [Test Case]

  1) Generate a self-signed certificate or use whatever existing SSL
  certificate already in place.

  If one want to create a PEM file for civetweb, instructions can be found here :
  https://github.com/civetweb/civetweb/blob/master/docs/OpenSSL.md

  ** Note: "CivetWeb requires one certificate file in PEM format" **

  2) Enable logging and debugging in "/etc/ceph/ceph.conf"

  Example:
  ------
  log to syslog = true
  err to syslog = true
  clog to syslog = true
  debug rgw = 10/5
  debug civetweb = 1/10
  ------

  http://docs.ceph.com/docs/mimic/rados/troubleshooting/log-and-debug/

  3) From the radosgw node, modify "/etc/ceph/ceph.conf" as follow:
  rgw_frontends = civetweb port=443s ssl_certificate=/<path_to_PEM_FILE>/<PEM_FILE>

  4) Restart the daemon:
  systemctl restart ceph-radosgw at rgw.`hostname -s`

  5) Look logs:
  2019-04-10 12:02:53.535133 7fcd20c4e000 0 civetweb: 0x562d710ed620: load_dll: libcrypto.so.1.1: cannot find CRYPTO_num_locks

  6) Look radosgw which should FAILED to start.
  systemctl status ceph-radosgw at rgw.`hostname -s`

  What we are looking for here is radosgw to be 'Active' and to have a
  LISTEN port on 443 as follow :

  $ netstat -anputa | grep LISTEN | grep 443 # or any port mentioned in the configuration above.
  tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      10153/radosgw

  [Potential Regression]

  * Same downgrade approach has been made for 'nodejs' via LP: #1798367

  * Nothing can be worst than current situation, considering that
  civetweb is non-functionnal when SSL is in used due to the
  incompatibility with 1.1 and make radosgw daemon to fail.

  * libssl1.0 and libssl1.1 are coinstallable ABIs so it shouldn't be a
  problem here.

  * See discussion IRC discussion (xnox/jamespage) on comment #11

  [Other Information]

  * Adding the OpenSSL 1.1 support has been explored and revealed to be non-trivial :
  https://github.com/civetweb/civetweb/pull/384/commits
  https://github.com/civetweb/civetweb/commit/adac9c916fa892ec5edce7b565803f1e62d304a2
  https://github.com/civetweb/civetweb/commit/5d83900fd29fb6fa1cd604676cb0562dc984dcc9

  http://docs.ceph.com/docs/bobtail/radosgw/troubleshooting/

  See discussion IRC discussion on comment #11

  [Original Description]

  Bionic's radosgw package (Version 12.2.11-0ubuntu0.18.04.1 ) can't run
  on Bionic, because the version of civetweb in Luminous is incompatible
  with libssl1.1, but it's built against libssl1.1.

  This has been known about upstream for a while now, and as noted in
  the bug-tracker (https://tracker.ceph.com/issues/20696), it can be
  fixed by building Luminous in an environment that has only libssl1.0
  available (or, in a more invasive manner, by incorporating a newer
  civetweb). A patch is in the tracker.ceph.com issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1822872/+subscriptions