[Bug 1594748] Re: CRYPTO_set_mem_functions() is broken

[Joy Latten]

Ok, this is also "broken" or an issue in upstream openssl 1.0.2 when OPENSSL_FIPS is defined. 
See, https://rt.openssl.org/Ticket/Display.html?id=4559#txn-68189 or
http://rt.openssl.org/Ticket/Display.html?id=4559

** Bug watch added: OpenSSL RT #4559
   http://rt.openssl.org/Ticket/Display.html?id=4559

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssl in Ubuntu.
https://bugs.launchpad.net/bugs/1594748

Title:
  CRYPTO_set_mem_functions() is broken

Status in openssl package in Ubuntu:
  New

Bug description:
  Description:	Ubuntu 16.04 LTS
  Release:	16.04

  openssl:
    Installed: 1.0.2g-1ubuntu4.1
    Candidate: 1.0.2g-1ubuntu4.1
    Version table:
   *** 1.0.2g-1ubuntu4.1 500
          500 http://fi.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
          500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
          100 /var/lib/dpkg/status
       1.0.2g-1ubuntu4 500
          500 http://fi.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

  CRYPTO_set_mem_functions() always returns 0 because library
  initialization already calls CRYPTO_malloc() and disables it:

  #0  CRYPTO_malloc (num=num at entry=1168, file=file at entry=0x7ffff70ae02c "fips_drbg_lib.c",
      line=line at entry=106) at mem.c:329
  #1  0x00007ffff70596df in FIPS_drbg_new (type=type at entry=0, flags=flags at entry=0)
      at fips_drbg_lib.c:106
  #2  0x00007ffff705aeb9 in FIPS_drbg_health_check (
      dctx=dctx at entry=0x7ffff731c960 <ossl_dctx>) at fips_drbg_selftest.c:760
  #3  0x00007ffff70595f0 in FIPS_drbg_init (dctx=dctx at entry=0x7ffff731c960 <ossl_dctx>,
      type=<optimized out>, flags=<optimized out>) at fips_drbg_lib.c:94
  #4  0x00007ffff6fe38f3 in RAND_init_fips () at rand_lib.c:287
  #5  0x00007ffff6f26f7a in OPENSSL_init_library () at o_init.c:119
  #6  0x00007ffff7de74ea in call_init (l=<optimized out>, argc=argc at entry=1,
      argv=argv at entry=0x7fffffffe5e8, env=env at entry=0x7fffffffe5f8) at dl-init.c:72
  #7  0x00007ffff7de75fb in call_init (env=0x7fffffffe5f8, argv=0x7fffffffe5e8, argc=1,
      l=<optimized out>) at dl-init.c:30
  #8  _dl_init (main_map=main_map at entry=0x640380, argc=1, argv=0x7fffffffe5e8,
      env=0x7fffffffe5f8) at dl-init.c:120

  This doesn't happen in upstream OpenSSL or in Debian's OpenSSL.
  Looking at the patches, this is caused by FIPS_drbg_init() in
  openssl-1.0.2g-fips.patch:

  +    if (!(dctx->xflags & DRBG_FLAG_TEST)) {
  +        if (!FIPS_drbg_health_check(dctx)) {
  +            FIPSerr(FIPS_F_FIPS_DRBG_INIT, FIPS_R_SELFTEST_FAILURE);
  +            return 0;
  +        }
  +    }

  I don't want any FIPS mode enabled though, so does it really even need
  to call RAND_init_fips() then?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1594748/+subscriptions