[Bug 1820333] Re: [SRU] ldap search should not encode attributes

Corey Bryant corey.bryant at canonical.com
Wed Mar 20 14:34:25 UTC 2019 

** Changed in: cloud-archive
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to keystone in Ubuntu.
https://bugs.launchpad.net/bugs/1820333

Title:
  [SRU] ldap search should not encode attributes

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in OpenStack Identity (keystone):
  Fix Released
Status in keystone package in Ubuntu:
  Fix Released
Status in keystone source package in Cosmic:
  Fix Committed

Bug description:
  [Impact]

  Listing user fails with LDAP backend fails
  ------------------------------------------

  $ openstack user list --debug --domain userdomain
  Request returned failure status: 400
  ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
  Traceback (most recent call last):
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
      result = cmd.run(parsed_args)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
      return super(Command, self).run(parsed_args)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
      column_names, data = self.take_action(parsed_args)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
      group=group,
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
      return wrapped(*args, **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
      **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
      return f(*args, **new_kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
      list_resp = self._list(url_query, self.collection_key)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
      resp, body = self.client.get(url, **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
      return self.request(url, 'GET', **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
      resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
      return self.session.request(url, method, **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
      raise exceptions.from_response(resp, method, url)
  keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
  clean_up ListUser: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
  Traceback (most recent call last):
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
      ret_val = super(OpenStackShell, self).run(argv)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 281, in run
      result = self.run_subcommand(remainder)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
      ret_value = super(OpenStackShell, self).run_subcommand(argv)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
      result = cmd.run(parsed_args)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
      return super(Command, self).run(parsed_args)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
      column_names, data = self.take_action(parsed_args)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
      group=group,
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
      return wrapped(*args, **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
      **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
      return f(*args, **new_kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
      list_resp = self._list(url_query, self.collection_key)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
      resp, body = self.client.get(url, **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
      return self.request(url, 'GET', **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
      resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
      return self.session.request(url, method, **kwargs)
    File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
      raise exceptions.from_response(resp, method, url)
  keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)

  END return value: 1

  /var/log/keystone/keystone.log
  ------------------------------
  (keystone.common.wsgi): 2019-03-15 15:26:15,385 ERROR ('attrs_from_List(): expected string in list', b'mail')
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__
      result = method(req, **params)
    File "/usr/lib/python3/dist-packages/keystone/common/controller.py", line 103, in wrapper
      return f(self, request, filters, **kwargs)
    File "/usr/lib/python3/dist-packages/keystone/identity/controllers.py", line 71, in list_users
      domain_scope=domain, hints=hints
    File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped
      __ret_val = __f(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 416, in wrapper
      return f(self, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 426, in wrapper
      return f(self, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1061, in list_users
      ref_list = self._handle_shadow_and_local_users(driver, hints)
    File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1044, in _handle_shadow_and_local_users
      return driver.list_users(hints) + fed_res
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 87, in list_users
      return self.user.get_all_filtered(hints)
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 327, in get_all_filtered
      for user in self.get_all(query, hints)]
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 319, in get_all
      hints=hints)
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1888, in get_all
      return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1590, in get_all
      for x in self._ldap_get_all(hints, ldap_filter)]
    File "/usr/lib/python3/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper
      return f(self, hints, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1543, in _ldap_get_all
      attrs)
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 976, in search_s
      attrlist_utf8, attrsonly)
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 654, in wrapper
      return func(self, conn, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 803, in search_s
      attrsonly)
    File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 858, in search_s
      return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
    File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1264, in search_ext_s
      return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
    File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1202, in _apply_method_s
      return func(self,*args,**kwargs)
    File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 851, in search_ext_s
      msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
    File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 847, in search_ext
      timeout,sizelimit,
    File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 319, in _ldap_call
      result = func(*args,**kwargs)
  TypeError: ('attrs_from_List(): expected string in list', b'mail')

  ---------------------------------------------

  In search_s() we're still encoding attrlist (note similar behavior in
  paged_search_s):

  attrlist_utf8 = list(map(utf8_encode, attrlist))

  Looking closer at the attribute list these all appear to be attribute
  names and that also appears to be how LDAP searches generally work;
  they specify attribute names they want to return, not values:

  [b'enabled', b'sn', b'userPassword', b'cn', b'description', b'mail']

  In Python 3 (and Python2 with bytes_mode=False) python-ldap no longer
  allows bytes for some fields (DNs, RDNs, attribute names, queries).
  Instead, text values are represented as str, the Unicode text type.

  A prior patch to Keystone's LDAP backend (see commit
  eca0829c4c65e6b64f08023ce2d5a55dc329248f) enabled this support but
  missed the above lines of code.

  Changing the above line of code to not utf8 encode the attrlist fixes
  the problem for me.

  [Test Case]

  Run charm-keystone-ldap functional tests for OpenStack Rocky or above.
  Upstream unit tests are also run.

  [Regression Potential]
  The only regression potential would be for PY2 code paths. PY3 code paths never worked for keystone's LDAP backend. The approach to the patch have purposefully minimized amount of code required and therefore regression potential for PY2. Note that Rocky for Ubuntu supports PY2 but as of Stein Ubuntu has dropped PY2 support.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1820333/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list