How to further handle Openssl 1.1.1 in Bionic?

Thu Oct 10 12:03:26 UTC 2019

On Thu, Oct 10, 2019 at 12:03 PM Dimitri John Ledkov <xnox at ubuntu.com> wrote:
>
> On Wed, 9 Oct 2019 at 17:41, Christian Ehrhardt
> <christian.ehrhardt at canonical.com> wrote:
> >
> > Hi,
> > in recent weeks since [1][2] there were quite some bugs related to
> > rebuilds or feature requests.
> > Those kind of issues seemed to be partially expected quoting the bugs SRU text:
> >
> > "OpenSSL 1.1.1 is ABI/API compatible with 1.1.0, however some software
> > is sensitive to the negotiation handshake and may either need
> > patches/improvements or clamp-down to maximum v1.2."
> >
> > Along the SRU some packages were initially identified needing patches
> > to either fully support (if doable in an SRU scope) or clamp-down to
> > TLSv1.2 and got such changes.
> >
> > But since then there is a set of bugs [3] coming up for either
> > a) "could you also enable TLSv1.3 in package ..."
> > b) "Openssl 1.1.1 broke package ..."
> >
>
> There is no generic guidance we can give on what to do with these, as
> they need to be on a case by case basis.
>
> I have submitted a few rebuilds were clearly the code is sensitive to
> openssl used at built time and picks up libssl1.1 >= 1.1.1 shared
> library dependency based on symbols used, and those got rejected by
> the SRU team.
>
> In general, the point of openssl 1.1.1 SRU was to make openssl
> supportable (maintainable, certifiable) over the Bionic timeframe. It
> was not to enable TLSv1.3 across the board.

I remember we talked about the reasoning before, thanks for the
detailed response and filling in the gaps that I had and those that I
might have forgotten :-)

> We did call the risk of connectivity issues. And those need to be
> handled on a case by case basis. SNI was explicitly called out for,
> and yes we had to fix about a dozen packages to do that. Arguably
> things should have been using SNI for years now, and it is a security
> improvement to verify, enforce, and use SNI properly. Thus when issue
> is w.r.t. SNI patching to support SNI is the correct course of action
> and usually simple to do.
>
> So far connectivity issues have been minor compared with when we
> enabled TLSv1.2, then had to disable it by default (but keep
> compiled), then enable it back in. And these days, we are luckly
> enough to have openssl.cnf system-wide way to set MaxProtocol=TLSv1.2
> to prevent TLSv1.3 based connectivity issues locally.
>
>
>
> > And while some of those seem to be "just work" e.g. a rebuild or small
> > patch to enable/disable TLSv1.3 others run into interesting issues as
> > openssl has changed more than jsut adding TLSv1.3.
> >
> > A good example is a bug that was expected to just be a rebuild [4]. We
> > have realized there can be subtle effects causing regressions. In the
> > particular example it seems that a rebuild not only enabled TLSv1.3,
> > but also bumped the minimum dh key size to 2k [5] which in turn breaks
> > some older clients and therefore is a no-no from an SRU perspective.
>
> That's not quite correct assessment of things. We will break people
> and will trade connectivity for better security. That's why we have
> disabled SSLv3, mitigated poodle attacks, etc. We will continue to
> require entropy, and higher key sizes, and better key-exchange
> algorithms as we go along. And sometimes that includes security
> updates, which SRUs build on top of. The change-effect you describe is
> due to a security update of openssl, which trumps SRUs. OpenSSL 1.1.0
> & 1.1.1 have raised a set of minimum key size requirements, mostly
> breaking all the test-suites with pre-generated tiny keys, but some
> real users too.
>
> As a local configuration, I believe one can lower OpenSSL security
> requirements by setting CipherString = DEFAULT at SECLEVEL=0 which will
> bring down requirements down to like pre 1.0.2, but that should only
> done as a local site configuration, and clients haunted down and
> upgraded/fixed.

Great suggestion, I'll give this a try somewhen later.

> For the HAPROXY, it would be nice to check if CipherString =
> DEFAULT at SECLEVEL=0 "helps" to bring down dh sizes to less secure ones,
> and if smaller dh sizes are pre-computed/available still. I would only
> call this is a regression, if there really is no way to use smaller dh
> sizes and if they are stopped being available at all.
>
> IMHO the haproxy SRU should not have been accepted, should now be
> tagged proposed-regression and removed from bionic-proposed. Which is
> a normal SRU process, and retry later. Or like discover some CVE and
> ask security team to deal with the rebuild =)))))
>
> > The bug [4] currently is assigned to ubuntu-security team for guidance
> > on this - do we want/need this and accept the regression it causes or
> > do we want/need to "clamp-down" the dh key size as well?
> >
>
> NAK
>
> > But this is a generic question - not only in the context of haproxy for [4].
> > If formerly only "clamping down for TLSv1.2" was considered, do we
> > need to revisit all packages for DH key size as well? ...
> >
>
> NAK
>
> > Even if we don't do anything today, a security update tomorrow might
> > force us to rebuild and trigger this kind of issues for 'potentially'
> > all dependencies of openssl.
> > We already have seen quite some of them being actual regressions in
> > our LTS which is concerning for the potential estimated number of
> > unreported cases.
> >
>
> That was a known risk, that's why we are choosing not to blindly
> rebuild things, especially at the far tail in universe.
>
> This openssl sru in bionic has uncovered broken packages, which were
> broken in cosmic, disco and eoan, despite shipping 1.1.1 there. It
> does indicate that the long tail of universe packages has less than
> adequate testing & autopkgtest & users in place.
>
> > And this is what this mail is about, as more than just bug-triagers
> > and the security-team might have a say and an opinion about it that
> > should be heard. Questions are:
> > - Are other packages known to likely could be affected by that?
> > - How was that planned from the POV of the openssl upload?
> >   What are we expected to do in the case of [4] or any similar issue
> > that we find later on?
> > - Do we need to analyze all packages rebuilt since openssl 1.1.1 for
> > such effects?
> > ...
> >
> > If you are involved or have context expertise please help to clarify
> > the questions above for the current issue [4] but also in general.
> > If not I'd still ask everyone to speak up if you know or have seen
> > related issues.
> > Triagers can use the tag "bionic-openssl-1.1" to help tracking those bugs.
> >
>
> The tag is a nice one, as it is expected for the number of issues that
> do come up to die down in frequency, and each case at this point will
> be unique enough requiring manual review / plan of action.
>
>
> > [1]: https://launchpad.net/ubuntu/+source/openssl/1.1.1-1ubuntu2.1~18.04.1
> > [2]: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1797386
> > [3]: https://bugs.launchpad.net/bugs/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.tag=bionic-openssl-1.1&orderby=status&start=0
> > [4]: https://bugs.launchpad.net/ubuntu/bionic/+source/haproxy/+bug/1841936
> > [5]: https://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-4000.html
> >
> > --
> > Christian Ehrhardt
> > Software Engineer, Ubuntu Server
>
> *Staff  ;-)
>
> --
> Regards,
>
> Dimitri.

-- 
Christian Ehrhardt
Staff Engineer, Ubuntu Server
Canonical Ltd