[Bug 1624856] Re: Regressions in GnuTLS 3.5.3 break OpenConnect

Fri Oct 27 01:04:52 UTC 2017

Launchpad has imported 15 comments from the remote bug at
https://bugzilla.redhat.com/show_bug.cgi?id=1370881.

If you reply to an imported comment from within Launchpad, your comment
will be sent to the remote bug automatically. Read more about
Launchpad's inter-bugtracker facilities at
https://help.launchpad.net/InterBugTracking.

------------------------------------------------------------------------
On 2016-08-27T20:55:59+00:00 Lars wrote:

Created attachment 1194946
verbose openconnect output (see end of file for failure)

Following the F25 update to gnutls-3.5.3-1, VPN connections established
through OpenConnect (against an AnyConnect VPN provided by a Cisco ASA
device) started to fail, with the remote side appearing dead to
OpenConnect:

DTLS Dead Peer Detection detected dead peer!

After re-establishment, the connection works again for some time, then
the process is repeated. The connection only fails if it's actually used
for something. Not quite sure yet what exactly triggers it. Logging into
a RHEV/oVirt web interface seems quite reliable, but I've also seen it
happen during DNF package installs.

Disabling DTLS (--no-dtls to openconnect) makes things work again, as
does downgrading of gnutls to the previous 3.5.2 version. Looking at the
upstream changelog, 3.5.3 appears to introduce a new DTLS sliding window
implementation, maybe related?

Version-Release number of selected component (if applicable):
gnutls-3.5.3-1.fc25.x86_64
openconnect-7.07-2.fc25.x86_64

Steps to Reproduce:
1. connect to AnyConnect VPN using OpenConnect
2. use it for some time (not sure what exactly triggers it, doesn't take long though)
3. connection dies with "DTLS Dead Peer Detection detected dead peer!"

Additional info:
I'm well aware that Cisco's DTLS implementation is quite non-standard, but grepping through the GnuTLS code, it seems to me that the intent is to support it (as DTLS0.9), which is why I'm filing this bug against GnuTLS.

OpenConnect does not show the issue when using gnutls-3.5.2-1.fc25 or
when built against OpenSSL.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/0

------------------------------------------------------------------------
On 2016-08-29T06:32:21+00:00 Nikos wrote:

Hi, if you downgrade to 3.5.2 does it work? Do you have any debugging
output when this issue happens?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/1

------------------------------------------------------------------------
On 2016-08-29T06:34:08+00:00 Nikos wrote:

The best would be if you could provide some gnutls debugging output
while that issue occurs. To get that you can set GNUTLS_DEBUG_LEVEL=6
environment variable prior to running openconnect.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/2

------------------------------------------------------------------------
On 2016-08-29T07:49:44+00:00 David wrote:

You can also add --gnutls-debug=6 to the openconnect command line.

It might be worth getting a packet capture from the real network,
showing the DTLS packets. The sequence numbers are in the clear, aren't
they? And the only obvious DTLS change I see between 3.5.2 and 3.5.3 is
related to the sequence number sliding window — so we hopefully don't
even *need* to decrypt the packets to work out what the problem is.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/3

------------------------------------------------------------------------
On 2016-08-29T08:19:52+00:00 Nikos wrote:

David reproduced the issue and send the log back. I'll try to figure the
issue from that.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/4

------------------------------------------------------------------------
On 2016-08-29T08:29:12+00:00 David wrote:

REC[0x15970d0]: Decrypted Packet[1.798] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.803] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.800] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.805] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.802] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.807] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.804] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.808] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.806] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.810] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.809] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.812] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.814] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.816] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.818] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.820] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.822] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.824] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.826] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.828] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.830] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.832] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.834] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.836] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.838] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.840] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.842]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.844]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.846]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.848]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.849]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.851]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.853]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.855]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.857]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.859]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.861]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.863]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.865]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.867]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.869]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.871]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.873]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.875]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.877]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.879]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.881]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.883]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.885]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.887]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.888]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.890]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.892]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.894]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.896]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.898]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.900]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.902]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.904]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.906]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.908]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.910]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.912]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.914]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.916]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.918]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.920]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.922]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.924]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.926]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.928]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.929]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.931]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.933]: Application Data
REC[0x15970d0]: Decrypted Packet[1.811] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.813] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.935]: Application Data
REC[0x15970d0]: Decrypted Packet[1.815] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.817] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.937]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.939]: Application Data
REC[0x15970d0]: Decrypted Packet[1.821] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.941]: Application Data
REC[0x15970d0]: Decrypted Packet[1.823] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.943]: Application Data
REC[0x15970d0]: Decrypted Packet[1.825] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.827] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.945]: Application Data
REC[0x15970d0]: Decrypted Packet[1.829] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.947]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.949]: Application Data
REC[0x15970d0]: Decrypted Packet[1.831] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.833] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.951]: Application Data
REC[0x15970d0]: Decrypted Packet[1.835] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.953]: Application Data
REC[0x15970d0]: Decrypted Packet[1.837] Application Data(23) with length: 1267
REC[0x15970d0]: Decrypted Packet[1.839] Application Data(23) with length: 1267
REC[0x15970d0]: Discarded duplicate message[1.955]: Application Data
REC[0x15970d0]: Discarded duplicate message[1.841]: Application Data
... and it never recovers...

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/5

------------------------------------------------------------------------
On 2016-08-29T11:53:12+00:00 Nikos wrote:

https://gitlab.com/gnutls/gnutls/commit/3ca24f24d9322256bc4c6d3bd237f8b98f073030

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/6

------------------------------------------------------------------------
On 2016-08-29T13:06:42+00:00 Fedora wrote:

gnutls-3.5.3-2.fc25 has been submitted as an update to Fedora 25.
https://bodhi.fedoraproject.org/updates/FEDORA-2016-66cab49b1f

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/7

------------------------------------------------------------------------
On 2016-08-29T16:31:58+00:00 David wrote:

Not sure I understand DTLS_WINDOW_RECEIVED() being conditional. Assuming
you only act on this when a packet passes the MAC validation, why do you
not *always* advance the window so that it ends at the latest-received
packet?

And if you are marking packets received before their MAC is checked, we
have bigger things to talk about...

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/8

------------------------------------------------------------------------
On 2016-08-29T22:55:32+00:00 Fedora wrote:

gnutls-3.5.3-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-66cab49b1f

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/9

------------------------------------------------------------------------
On 2016-08-30T04:35:42+00:00 Nikos wrote:

(In reply to David Woodhouse from comment #8)
> Not sure I understand DTLS_WINDOW_RECEIVED() being conditional. Assuming you
> only act on this when a packet passes the MAC validation, why do you not
> *always* advance the window so that it ends at the latest-received packet? 

What if first packet is 1 and you receive 3, 6, 5, 4, 2. If window is
advanced to the latest 5, 4, 2 would be discarded.

> And if you are marking packets received before their MAC is checked, we have
> bigger things to talk about...

That's after MAC is checked.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/10

------------------------------------------------------------------------
On 2016-08-30T09:53:47+00:00 David wrote:

(In reply to Nikos Mavrogiannopoulos from comment #10)
> (In reply to David Woodhouse from comment #8)
> > Not sure I understand DTLS_WINDOW_RECEIVED() being conditional. Assuming you
> > only act on this when a packet passes the MAC validation, why do you not
> > *always* advance the window so that it ends at the latest-received packet? 
> 
> What if first packet is 1 and you receive 3, 6, 5, 4, 2. If window is
> advanced to the latest 5, 4, 2 would be discarded.

The way this usually works is the window advances so that the *END* of
the window is the last-received packet.

So the *end* of the window is advanced to the latest 6. The bitmap then
represents the packets leading up to that.

cf.
http://git.infradead.org/users/dwmw2/openconnect.git/blob/HEAD:/esp.c#l33

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/11

------------------------------------------------------------------------
On 2016-08-30T14:39:48+00:00 David wrote:

Created attachment 1195938
Import sliding window code from OpenConnect

Here's my code from OpenConnect. We *always* advance the sliding window
so it ends at the latest received packet. We never discard a packet
because it's "too new".

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/12

------------------------------------------------------------------------
On 2016-08-31T11:31:03+00:00 David wrote:

Updated patch in https://gitlab.com/gnutls/gnutls/merge_requests/55

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/13

------------------------------------------------------------------------
On 2016-09-06T18:23:53+00:00 Fedora wrote:

gnutls-3.5.3-2.fc25 has been pushed to the Fedora 25 stable repository.
If problems still persist, please make note of it in this bug report.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/comments/14

** Changed in: gnutls (Fedora)
       Status: Unknown => Fix Released

** Changed in: gnutls (Fedora)
   Importance: Unknown => Undecided

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1624856

Title:
  Regressions in GnuTLS 3.5.3 break OpenConnect

Status in gnutls28 package in Ubuntu:
  Fix Released
Status in gnutls28 package in Debian:
  Fix Released
Status in gnutls package in Fedora:
  Fix Released

Bug description:
  OpenConnect is unusable after upgrading to GnuTLS 3.5.3: the VPN
  connection drops after less than a minute.  Please merge gnutls28
  3.5.3-5 or 3.5.4-2 from Debian, where this problem was fixed
  (https://bugs.debian.org/835587).

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1624856/+subscriptions