[Bug 1833039] Re: 18.04/Apache2: rejecting client initiated renegotiation due to openssl 1.1.1

Andreas Hasenack andreas at canonical.com
Mon Jul 1 17:47:03 UTC 2019


** Description changed:

  [Impact]
  Under the following conditions, https connections using client cert authentication will suffer a long delay (about 15s if modreqtimeout is enabled, more if it is disabled):
  * TLSv1.2
  * client certificate authentication in use
  * a Location, Directory, or other such block defining the client certificate authentication for that block only, differing from the SSL vhost as a whole
  
  This was triggered by the OpenSSL 1.1.1 SRU and was caused by this
  openssl change in SSL_MODE_AUTO_RETRY from disabled to enabled by
  default:
  https://github.com/openssl/openssl/blob/a4a90a8a3bdcb9336b5c9c15da419e99a87bc6ed/CHANGES#L121-L130
  
  [Test Case]
  
  It helps if you have lxd up and running. Otherwise, a VM or even bare
  metal host also works, as long as you stick to the "ubuntu" hostname.
  
  Launch a container for the release you are testing. The command below is for bionic:
  $ lxc launch ubuntu-daily:bionic ubuntu
  
  Enter the container as root:
  $ lxc exec ubuntu bash
  
  Verify hostname is "ubuntu":
  # hostname
  ubuntu
  
  Install apache2:
  apt update && apt install apache2
  
  Download the following files from this bug report and place them in /etc/apache2:
  cd /etc/apache2
  wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274492/+files/cacert.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274493/+files/ubuntu.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274494/+files/ubuntu.key
  
  Adjust permissions of the key file:
  chmod 0640 /etc/apache2/ubuntu.key
  chgrp www-data /etc/apache2/ubuntu.key
  
  Download the client certificate and key files and place them in /root:
  cd /root
  wget https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274495/+files/client-auth.pem https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+attachment/5274496/+files/client-auth.key
- client-auth.key
- client-auth.pem
  
  Create this vhost file (caution, lines may wrap, in particular LogFormat: it should be one long line):
  cat > /etc/apache2/sites-available/cert-auth-test.conf <<EOF
  <IfModule mod_ssl.c>
      <VirtualHost _default_:443>
          LogLevel info ssl:warn
          ServerAdmin webmaster at localhost
          DocumentRoot /var/www/html
          LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" protocol=%{SSL_PROTOCOL}x commonName=%{SSL_CLIENT_S_DN_CN}x" combined-ssl
          ErrorLog \${APACHE_LOG_DIR}/error.log
          CustomLog \${APACHE_LOG_DIR}/access.log combined-ssl
          SSLEngine on
          SSLCertificateFile  /etc/apache2/ubuntu.pem
          SSLCertificateKeyFile /etc/apache2/ubuntu.key
          SSLCACertificateFile /etc/apache2/cacert.pem
          <FilesMatch "\.(cgi|shtml|phtml|php)$">
                  SSLOptions +StdEnvVars
          </FilesMatch>
          <Directory /usr/lib/cgi-bin>
                  SSLOptions +StdEnvVars
          </Directory>
          <Location />
                  SSLVerifyClient require
                  Require ssl-verify-client
          </Location>
      </VirtualHost>
  </IfModule>
  EOF
  
  Enable the ssl module and this new vhost we just created:
  a2enmod ssl && a2ensite cert-auth-test.conf
  
  Restart apache2:
  systemctl restart apache2
  
  If at this stage you try the following command, it will fail like this because no client certificate was provided:
  # curl --output /dev/null https://ubuntu/ --cacert /etc/apache2/cacert.pem
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  curl: (56) OpenSSL SSL_read: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure, errno 0
  
  And the apache error log will confirm the reason:
  [Mon Jul 01 14:10:23.312645 2019] [ssl:error] [pid 1685:tid 140326396421888] SSL Library Error: error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate -- No CAs known to server for verification?
  
  Now retry, but providing the client certificate and key files, and forcing TLSv1.2 just to be sure. Due to the bug, the command will stall for about 15 seconds, but the index.html file will be downloaded:
  # rm -f index.html
  # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 10918  100 10918    0     0    706      0  0:00:15  0:00:15 --:--:--  2579
  # ll index.html
  -rw-r--r-- 1 root root 10918 Jul  1 14:15 index.html
  
  Apache will log this in the error.log file:
  [Mon Jul 01 14:15:38.014784 2019] [reqtimeout:info] [pid 1685:tid 140326278772480] [client 10.0.100.215:35108] AH01382: Request body read timeout
  
  That is due to modreqtimeout kicking in.
  In the access.log file, we will have the request:
  10.0.100.215 - - [01/Jul/2019:14:15:22 +0000] "GET / HTTP/1.1" 200 16544 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth
  
  The protocol and commonName parts confirm the protocol that was used, and the commonName of the client certificate that was used for authentication.
  So it works, but takes a long time for each request. This verifies the bug.
  
  After installing the fixed apache2 packages, the download completes almost instantly:
  # curl --output index.html https://ubuntu/ --cacert /etc/apache2/cacert.pem --cert client-auth.pem --key client-auth.key --tlsv1.2
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 10918  100 10918    0     0   333k      0 --:--:-- --:--:-- --:--:--  333k
  
  The apache access log confirms the protocol and that client certificate authentication was used:
  10.0.100.215 - - [01/Jul/2019:14:29:56 +0000] "GET / HTTP/1.1" 200 16525 "-" "curl/7.58.0" protocol=TLSv1.2 commonName=client-auth
  
  And the error log gets no new entries. This verifies the bug is fixed.
  
  [Regression Potential]
  This is reverting, in mod_ssl, a settings change that was made in openssl 1.1.1. It's committed upstream in mod_ssl, and I found no other follow-up commits about this.
  This being SSL-related, of course it can have surprises and be complicated. The openssl commit even warns that hangs could occur (which happened in this bug here), and it's expected that applications that are affected adjust accordingly.
  
  [Other Info]
  The upstream mod_ssl commit that made this change also changed something else in the code, but we decided to not adopt it because it's TLSv1.3 specific, and that version of the protocol is not enabled in the apache builds from bionic or cosmic. TLSv1.3 is deemed complete only in apache 2.4.37 and later (http://www.apache.org/dist/httpd/CHANGES_2.4.37)
  
  [Original Description]
  I am using Apache2 with client certificate authentication.
  Since recently (last week) and without any configuration changes, the following errors occur frequently:
  
  AH02042: rejecting client initiated renegotiation
  
  Client connections are very slow and sometimes it takes more than a minute until a weg page can be opened in the browser.
  Before installation of the latest security fixes last week, this error did not occur.
  
  Could it be related to
  https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1803689?
  
  System information:
  
  Description:    Ubuntu 18.04.2 LTS
  Release:        18.04
  
  apache2:
    Installiert:           2.4.29-1ubuntu4.6
    Installationskandidat: 2.4.29-1ubuntu4.6
    Versionstabelle:
   *** 2.4.29-1ubuntu4.6 500
          500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
          500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
          100 /var/lib/dpkg/status
       2.4.29-1ubuntu4 500
          500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
  
  openssl:
    Installiert:           1.1.1-1ubuntu2.1~18.04.2
    Installationskandidat: 1.1.1-1ubuntu2.1~18.04.2
    Versionstabelle:
   *** 1.1.1-1ubuntu2.1~18.04.2 500
          500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
          100 /var/lib/dpkg/status
       1.1.0g-2ubuntu4.3 500
          500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
       1.1.0g-2ubuntu4 500
          500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

-- 
You received this bug notification because you are a member of Ubuntu
Server, which is subscribed to apache2 in Ubuntu.
https://bugs.launchpad.net/bugs/1833039

Title:
  18.04/Apache2: rejecting client initiated renegotiation due to openssl
  1.1.1

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039/+subscriptions



More information about the Ubuntu-server-bugs mailing list