[Bug 1906627] Re: GSS-SPNEGO implementation in cyrus-sasl2 is incompatible with Active Directory, causing recent adcli regression

Matthew Ruffell 1906627 at bugs.launchpad.net
Sun Dec 6 23:51:50 UTC 2020 

** Summary changed:

- adcli fails, can't contact LDAP server
+ GSS-SPNEGO implementation in cyrus-sasl2 is incompatible with Active Directory, causing recent adcli regression

** Description changed:

- Package: adcli
- Version: 0.8.2-1ubuntu1
- Release: Ubuntu 18.04 LTS
+ [Impact]
  
- When trying to join the domain with this new version of adcli, it gets
- to the point of 'Using GSS-SPNEGO for SASL bind' and then it will not do
- anything for 10 minutes. It will then fail, complaining it can't reach
- the LDAP server.
+ A recent release of adcli 0.8.2-1ubuntu1 to bionic-updates caused a
+ regression for some users when attempting to join a Active Directory
+ realm. adcli introduced a default behaviour change, moving from GSS-API
+ to GSS-SPNEGO as the default channel encryption algorithm.
  
- Logs:
- Dec 03 01:39:50 example001.domain.com realmd[6419]:  * Authenticated as user: domain-join-account at DOMAIN.COM
- Dec 03 01:39:50 example001.domain.com adcli[6459]: GSSAPI client step 1
- Dec 03 01:39:50 example001.domain.com realmd[6419]:  * Authenticated as user: domain-join-account at DOMAIN.COM
- Dec 03 01:39:50 example001.domain.com adcli[6459]: GSSAPI client step 1
- Dec 03 01:39:50 example001.domain.com realmd[6419]:  * Using GSS-SPNEGO for SASL bind
- Dec 03 01:39:50 example001.domain.com realmd[6419]:  * Using GSS-SPNEGO for SASL bind
- Dec 03 01:39:50 example001.domain.com adcli[6459]: GSSAPI client step 1
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  ! Couldn't lookup domain short name: Can't contact LDAP server
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  ! Couldn't lookup domain short name: Can't contact LDAP server
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using fully qualified name: example001.domain.com
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using fully qualified name: example001.domain.com
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using domain name: domain.com
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using domain name: domain.com
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using computer account name: EXAMPLE001
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using computer account name: EXAMPLE001
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using domain realm: domain.com
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using domain realm: domain.com
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Calculated computer account name from fqdn: EXAMPLE001
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Calculated computer account name from fqdn: EXAMPLE001
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * With user principal: host/example001.domain.com at DOMAIN.COM
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * With user principal: host/example001.domain.com at DOMAIN.COM
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Generated 120 character computer password
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Generated 120 character computer password
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using keytab: FILE:/etc/krb5.keytab
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  * Using keytab: FILE:/etc/krb5.keytab
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  ! Couldn't lookup computer account: EXAMPLE001$: Can't contact LDAP server
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  ! Couldn't lookup computer account: EXAMPLE001$: Can't contact LDAP server
- Dec 03 01:55:27 example001.domain.com realmd[6419]: adcli: joining domain domain.com failed: Couldn't lookup computer account: EXAMPLE001$: Can't contact LDAP server
- Dec 03 01:55:27 example001.domain.com realmd[6419]: adcli: joining domain domain.com failed: Couldn't lookup computer account: EXAMPLE001$: Can't contact LDAP server
- Dec 03 01:55:27 example001.domain.com realmd[6419]: process exited: 6459
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  ! Failed to join the domain
- Dec 03 01:55:27 example001.domain.com realmd[6419]:  ! Failed to join the domain
+ adcli uses the GSS-SPNEGO implementation from libsasl2-modules-gssapi-
+ mit, a part of cyrus-sasl2. The implementation seems to have some
+ compatibility issues with particular configurations of Active Directory
+ on recent Windows Server systems.
  
- On the network level, adcli gets to the point of send an ldap query to
- the domain controller and the domain controller returns an ack tcp
- packet, but then there is no more traffic between the domain controller
- and the server except for ntp packets until it fails.
+ Particularly, adcli sends a ldap query to the domain controller, which
+ responds with a tcp ack, but never returns a ldap response. The
+ connection just hangs at this point and no more traffic is sent.
  
- The domain controller traffic also shows that it is receiving the ldap
- query packet from the server but it never sends a reply and there is no
- log in directory services regarding the query. We also couldn't find
- anything in procmon regarding this query either.
+ You can see it on the packet trace below:
  
- Workaround/Fix:
- Downgrading the adcli package back to version 0.8.2-1 fixes the issues and domain join works properly again.
+ https://paste.ubuntu.com/p/WRnnRMGBPm/
+ 
+ On Focal, where the implementation of GSS-SPNEGO is working, we see a
+ full exchange, and adcli works as expected:
+ 
+ https://paste.ubuntu.com/p/8668pJrr2m/
+ 
+ The fix is to not assume use of confidentiality and integrity modes, and
+ instead use the flags negotiated by GSS-API during the initial
+ handshake, as required by Microsoft's implementation.
+ 
+ [Testcase]
+ 
+ You will need to set up a Windows Server 2019 system, install and
+ configure Active Directory and enable LDAP extensions and configure
+ LDAPS and import the AD SSL certificate to the Ubuntu client. Create
+ some users in Active Directory.
+ 
+ On the Ubuntu client, set up /etc/hosts with the hostname of the Windows
+ Server machine, if your system isn't configured for AD DNS.
+ 
+ From there, install adcli 0.8.2-1 from -release.
+ 
+ $ sudo apt install adcli
+ 
+ Set up a packet trace with tcpdump:
+ 
+ $ sudo tcpdump -i any port '(389 or 3268 or 636 or 3269)'
+ 
+ Next, join the AD realm using the normal GSS-API:
+ 
+ # adcli join --verbose -U Administrator --domain WIN-
+ SB6JAS7PH22.testing.local --domain-controller WIN-
+ SB6JAS7PH22.testing.local --domain-realm TESTING.LOCAL
+ 
+ You will be prompted for Administrator's passowrd.
+ 
+ The output should look like the below:
+ 
+ https://paste.ubuntu.com/p/NWHGQn746D/
+ 
+ Next, enable -proposed, and install adcli 0.8.2-1ubuntu1 which caused the regression.
+ Repeat the above steps. Now you should see the connection hang.
+ 
+ https://paste.ubuntu.com/p/WRnnRMGBPm/
+ 
+ Finally, install the fixed cyrus-sasl2 package, which is available from the
+ below ppa:
+ 
+ https://launchpad.net/~mruffell/+archive/ubuntu/lp1906627-test
+ 
+ $ sudo add-apt-repository ppa:mruffell/lp1906627-test
+ $ sudo apt-get update
+ $ sudo apt install libsasl2-2 libsasl2-modules libsasl2-modules-db libsasl2-modules-gssapi-mit
+ 
+ Repeat the steps. GSS-SPNEGO should be working as intended, and you
+ should get output like below:
+ 
+ https://paste.ubuntu.com/p/W5cJNGvCsx/
+ 
+ [Where problems could occur]
+ 
+ Since we are changing the implementation of GSS-SPNEGO, and cyrus-sasl2
+ is the library which provides it, we can potentially break any package
+ which depends on libsasl2-modules-gssapi-mit for GSS-SPNEGO.
+ 
+ $ apt rdepends libsasl2-modules-gssapi-mit
+ libsasl2-modules-gssapi-mit
+ Reverse Depends:
+  |Suggests: ldap-utils
+   Depends: adcli
+   Conflicts: libsasl2-modules-gssapi-heimdal
+  |Suggests: libsasl2-modules
+   Conflicts: libsasl2-modules-gssapi-heimdal
+  |Recommends: sssd-krb5-common
+  |Suggests: slapd
+  |Suggests: libsasl2-modules
+  |Suggests: ldap-utils
+  |Depends: msktutil
+   Conflicts: libsasl2-modules-gssapi-heimdal
+  |Depends: libapache2-mod-webauthldap
+   Depends: freeipa-server
+   Depends: freeipa-client
+   Depends: adcli
+   Depends: 389-ds-base
+  |Recommends: sssd-krb5-common
+  |Suggests: slapd
+  |Suggests: libsasl2-modules
+  
+ While this SRU makes cyrus-sasl2 work with Microsoft implementations of GSS-SPNEGO, which will be the more common usecase, it may change the behaviour  when connecting to a MIT krb5 server with the GSS-SPNEGO protocol, as krb5 assumes use of confidentiality and integrity modes. This shouldn't be a problem as the krb5 implementation signals its intentions by setting the correct flags during handshake, which these patches to cyrus-sasl2 should now parse correctly.
+ 
+ [Other Info]
+ 
+ The below two commits are needed. The first fixes the problem, the second fixes
+ some unused parameter warnings.
+ 
+ commit 816e529043de08f3f9dcc4097380de39478b0b16
+ Author: Simo Sorce <simo at redhat.com>
+ Date:   Thu Feb 16 15:25:56 2017 -0500
+ Subject: Fix GSS-SPNEGO mechanism's incompatible behavior
+ https://github.com/cyrusimap/cyrus-sasl/commit/816e529043de08f3f9dcc4097380de39478b0b16
+ 
+ commit ed2ad48f242fe16e846a9db552a04fca1a5da45f
+ Author: Simo Sorce <simo at redhat.com>
+ Date:   Tue Apr 11 18:31:46 2017 -0400
+ Subject: Drop unused parameter from gssapi_spnego_ssf()
+ https://github.com/cyrusimap/cyrus-sasl/commit/ed2ad48f242fe16e846a9db552a04fca1a5da45f

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1906627

Title:
  GSS-SPNEGO implementation in cyrus-sasl2 is incompatible with Active
  Directory, causing recent adcli regression

Status in adcli package in Ubuntu:
  Fix Released
Status in cyrus-sasl2 package in Ubuntu:
  Fix Released
Status in adcli source package in Bionic:
  In Progress
Status in cyrus-sasl2 source package in Bionic:
  In Progress

Bug description:
  [Impact]

  A recent release of adcli 0.8.2-1ubuntu1 to bionic-updates caused a
  regression for some users when attempting to join a Active Directory
  realm. adcli introduced a default behaviour change, moving from GSS-
  API to GSS-SPNEGO as the default channel encryption algorithm.

  adcli uses the GSS-SPNEGO implementation from libsasl2-modules-gssapi-
  mit, a part of cyrus-sasl2. The implementation seems to have some
  compatibility issues with particular configurations of Active
  Directory on recent Windows Server systems.

  Particularly, adcli sends a ldap query to the domain controller, which
  responds with a tcp ack, but never returns a ldap response. The
  connection just hangs at this point and no more traffic is sent.

  You can see it on the packet trace below:

  https://paste.ubuntu.com/p/WRnnRMGBPm/

  On Focal, where the implementation of GSS-SPNEGO is working, we see a
  full exchange, and adcli works as expected:

  https://paste.ubuntu.com/p/8668pJrr2m/

  The fix is to not assume use of confidentiality and integrity modes,
  and instead use the flags negotiated by GSS-API during the initial
  handshake, as required by Microsoft's implementation.

  [Testcase]

  You will need to set up a Windows Server 2019 system, install and
  configure Active Directory and enable LDAP extensions and configure
  LDAPS and import the AD SSL certificate to the Ubuntu client. Create
  some users in Active Directory.

  On the Ubuntu client, set up /etc/hosts with the hostname of the
  Windows Server machine, if your system isn't configured for AD DNS.

  From there, install adcli 0.8.2-1 from -release.

  $ sudo apt install adcli

  Set up a packet trace with tcpdump:

  $ sudo tcpdump -i any port '(389 or 3268 or 636 or 3269)'

  Next, join the AD realm using the normal GSS-API:

  # adcli join --verbose -U Administrator --domain WIN-
  SB6JAS7PH22.testing.local --domain-controller WIN-
  SB6JAS7PH22.testing.local --domain-realm TESTING.LOCAL

  You will be prompted for Administrator's passowrd.

  The output should look like the below:

  https://paste.ubuntu.com/p/NWHGQn746D/

  Next, enable -proposed, and install adcli 0.8.2-1ubuntu1 which caused the regression.
  Repeat the above steps. Now you should see the connection hang.

  https://paste.ubuntu.com/p/WRnnRMGBPm/

  Finally, install the fixed cyrus-sasl2 package, which is available from the
  below ppa:

  https://launchpad.net/~mruffell/+archive/ubuntu/lp1906627-test

  $ sudo add-apt-repository ppa:mruffell/lp1906627-test
  $ sudo apt-get update
  $ sudo apt install libsasl2-2 libsasl2-modules libsasl2-modules-db libsasl2-modules-gssapi-mit

  Repeat the steps. GSS-SPNEGO should be working as intended, and you
  should get output like below:

  https://paste.ubuntu.com/p/W5cJNGvCsx/

  [Where problems could occur]

  Since we are changing the implementation of GSS-SPNEGO, and cyrus-
  sasl2 is the library which provides it, we can potentially break any
  package which depends on libsasl2-modules-gssapi-mit for GSS-SPNEGO.

  $ apt rdepends libsasl2-modules-gssapi-mit
  libsasl2-modules-gssapi-mit
  Reverse Depends:
   |Suggests: ldap-utils
    Depends: adcli
    Conflicts: libsasl2-modules-gssapi-heimdal
   |Suggests: libsasl2-modules
    Conflicts: libsasl2-modules-gssapi-heimdal
   |Recommends: sssd-krb5-common
   |Suggests: slapd
   |Suggests: libsasl2-modules
   |Suggests: ldap-utils
   |Depends: msktutil
    Conflicts: libsasl2-modules-gssapi-heimdal
   |Depends: libapache2-mod-webauthldap
    Depends: freeipa-server
    Depends: freeipa-client
    Depends: adcli
    Depends: 389-ds-base
   |Recommends: sssd-krb5-common
   |Suggests: slapd
   |Suggests: libsasl2-modules
   
  While this SRU makes cyrus-sasl2 work with Microsoft implementations of GSS-SPNEGO, which will be the more common usecase, it may change the behaviour  when connecting to a MIT krb5 server with the GSS-SPNEGO protocol, as krb5 assumes use of confidentiality and integrity modes. This shouldn't be a problem as the krb5 implementation signals its intentions by setting the correct flags during handshake, which these patches to cyrus-sasl2 should now parse correctly.

  [Other Info]

  The below two commits are needed. The first fixes the problem, the second fixes
  some unused parameter warnings.

  commit 816e529043de08f3f9dcc4097380de39478b0b16
  Author: Simo Sorce <simo at redhat.com>
  Date:   Thu Feb 16 15:25:56 2017 -0500
  Subject: Fix GSS-SPNEGO mechanism's incompatible behavior
  https://github.com/cyrusimap/cyrus-sasl/commit/816e529043de08f3f9dcc4097380de39478b0b16

  commit ed2ad48f242fe16e846a9db552a04fca1a5da45f
  Author: Simo Sorce <simo at redhat.com>
  Date:   Tue Apr 11 18:31:46 2017 -0400
  Subject: Drop unused parameter from gssapi_spnego_ssf()
  https://github.com/cyrusimap/cyrus-sasl/commit/ed2ad48f242fe16e846a9db552a04fca1a5da45f

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/+subscriptions

More information about the Ubuntu-sponsors mailing list