How to further handle Openssl 1.1.1 in Bionic?

Thu Oct 10 10:03:43 UTC 2019

On Wed, 9 Oct 2019 at 17:41, Christian Ehrhardt
<christian.ehrhardt at canonical.com> wrote:
>
> Hi,
> in recent weeks since [1][2] there were quite some bugs related to
> rebuilds or feature requests.
> Those kind of issues seemed to be partially expected quoting the bugs SRU text:
>
> "OpenSSL 1.1.1 is ABI/API compatible with 1.1.0, however some software
> is sensitive to the negotiation handshake and may either need
> patches/improvements or clamp-down to maximum v1.2."
>
> Along the SRU some packages were initially identified needing patches
> to either fully support (if doable in an SRU scope) or clamp-down to
> TLSv1.2 and got such changes.
>
> But since then there is a set of bugs [3] coming up for either
> a) "could you also enable TLSv1.3 in package ..."
> b) "Openssl 1.1.1 broke package ..."
>

There is no generic guidance we can give on what to do with these, as
they need to be on a case by case basis.

I have submitted a few rebuilds were clearly the code is sensitive to
openssl used at built time and picks up libssl1.1 >= 1.1.1 shared
library dependency based on symbols used, and those got rejected by
the SRU team.

In general, the point of openssl 1.1.1 SRU was to make openssl
supportable (maintainable, certifiable) over the Bionic timeframe. It
was not to enable TLSv1.3 across the board.

We did call the risk of connectivity issues. And those need to be
handled on a case by case basis. SNI was explicitly called out for,
and yes we had to fix about a dozen packages to do that. Arguably
things should have been using SNI for years now, and it is a security
improvement to verify, enforce, and use SNI properly. Thus when issue
is w.r.t. SNI patching to support SNI is the correct course of action
and usually simple to do.

So far connectivity issues have been minor compared with when we
enabled TLSv1.2, then had to disable it by default (but keep
compiled), then enable it back in. And these days, we are luckly
enough to have openssl.cnf system-wide way to set MaxProtocol=TLSv1.2
to prevent TLSv1.3 based connectivity issues locally.

> And while some of those seem to be "just work" e.g. a rebuild or small
> patch to enable/disable TLSv1.3 others run into interesting issues as
> openssl has changed more than jsut adding TLSv1.3.
>
> A good example is a bug that was expected to just be a rebuild [4]. We
> have realized there can be subtle effects causing regressions. In the
> particular example it seems that a rebuild not only enabled TLSv1.3,
> but also bumped the minimum dh key size to 2k [5] which in turn breaks
> some older clients and therefore is a no-no from an SRU perspective.

That's not quite correct assessment of things. We will break people
and will trade connectivity for better security. That's why we have
disabled SSLv3, mitigated poodle attacks, etc. We will continue to
require entropy, and higher key sizes, and better key-exchange
algorithms as we go along. And sometimes that includes security
updates, which SRUs build on top of. The change-effect you describe is
due to a security update of openssl, which trumps SRUs. OpenSSL 1.1.0
& 1.1.1 have raised a set of minimum key size requirements, mostly
breaking all the test-suites with pre-generated tiny keys, but some
real users too.

As a local configuration, I believe one can lower OpenSSL security
requirements by setting CipherString = DEFAULT at SECLEVEL=0 which will
bring down requirements down to like pre 1.0.2, but that should only
done as a local site configuration, and clients haunted down and
upgraded/fixed.

For the HAPROXY, it would be nice to check if CipherString =
DEFAULT at SECLEVEL=0 "helps" to bring down dh sizes to less secure ones,
and if smaller dh sizes are pre-computed/available still. I would only
call this is a regression, if there really is no way to use smaller dh
sizes and if they are stopped being available at all.

IMHO the haproxy SRU should not have been accepted, should now be
tagged proposed-regression and removed from bionic-proposed. Which is
a normal SRU process, and retry later. Or like discover some CVE and
ask security team to deal with the rebuild =)))))

> The bug [4] currently is assigned to ubuntu-security team for guidance
> on this - do we want/need this and accept the regression it causes or
> do we want/need to "clamp-down" the dh key size as well?
>

NAK

> But this is a generic question - not only in the context of haproxy for [4].
> If formerly only "clamping down for TLSv1.2" was considered, do we
> need to revisit all packages for DH key size as well? ...
>

NAK

> Even if we don't do anything today, a security update tomorrow might
> force us to rebuild and trigger this kind of issues for 'potentially'
> all dependencies of openssl.
> We already have seen quite some of them being actual regressions in
> our LTS which is concerning for the potential estimated number of
> unreported cases.
>

That was a known risk, that's why we are choosing not to blindly
rebuild things, especially at the far tail in universe.

This openssl sru in bionic has uncovered broken packages, which were
broken in cosmic, disco and eoan, despite shipping 1.1.1 there. It
does indicate that the long tail of universe packages has less than
adequate testing & autopkgtest & users in place.

> And this is what this mail is about, as more than just bug-triagers
> and the security-team might have a say and an opinion about it that
> should be heard. Questions are:
> - Are other packages known to likely could be affected by that?
> - How was that planned from the POV of the openssl upload?
>   What are we expected to do in the case of [4] or any similar issue
> that we find later on?
> - Do we need to analyze all packages rebuilt since openssl 1.1.1 for
> such effects?
> ...
>
> If you are involved or have context expertise please help to clarify
> the questions above for the current issue [4] but also in general.
> If not I'd still ask everyone to speak up if you know or have seen
> related issues.
> Triagers can use the tag "bionic-openssl-1.1" to help tracking those bugs.
>

The tag is a nice one, as it is expected for the number of issues that
do come up to die down in frequency, and each case at this point will
be unique enough requiring manual review / plan of action.

> [1]: https://launchpad.net/ubuntu/+source/openssl/1.1.1-1ubuntu2.1~18.04.1
> [2]: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1797386
> [3]: https://bugs.launchpad.net/bugs/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.tag=bionic-openssl-1.1&orderby=status&start=0
> [4]: https://bugs.launchpad.net/ubuntu/bionic/+source/haproxy/+bug/1841936
> [5]: https://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-4000.html
>
> --
> Christian Ehrhardt
> Software Engineer, Ubuntu Server

*Staff  ;-)

-- 
Regards,

Dimitri.