[Bug 1850634] Re: queens regresion: _dn_to_id() not using utf8_encode/decode
Felipe Reyes
1850634 at bugs.launchpad.net
Mon Jan 13 22:13:41 UTC 2020
verified xenial-queens, no regressions detected, testing journal:
$ time tox -e func-smoke
func-smoke installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support,amulet==1.21.0,aodhclient==1.5.0,appdirs==1.4.3,Babel==2.8.0,backports.os==0.1.1,blessings==1.6,bundletester==0.12.2,certifi==2019.11.28,cffi==1.13.2,chardet==3.0.4,charm-tools==2.7.2,charmhelpers==0.20.7,Cheetah3==3.2.4,cliff==2.18.0,cmd2==0.8.9,colander==1.7.0,configparser==4.0.2,contextlib2==0.6.0.post1,coverage==5.0.3,cryptography==2.8,debtcollector==1.22.0,decorator==4.4.1,dict2colander==0.2,distro==1.4.0,distro-info==0.0.0,dogpile.cache==0.9.0,entrypoints==0.3,enum34==1.1.6,extras==1.0.0,fasteners==0.15,fixtures==3.0.0,flake8==2.4.1,funcsigs==1.0.2,functools32==3.2.3.post2,future==0.18.2,futures==3.3.0,futurist==1.10.0,gnocchiclient==3.1.1,httplib2==0.15.0,idna==2.8,importlib-metadata==1.4.0,ipaddress==1.0.23,iso8601==0.1.12,Jinja2==2.10.3,jmespath==0.9.4,jsonpatch==1.24,jsonpointer==2.0,jsonschema==2.5.1,juju-deployer==0.11.0,juju-wait==2.5.0,jujubundlelib==0.5.6,jujuclient==0.54.0,keyring==18.0.1,keystoneauth1==3.18.0,launchpadlib==1.10.9,lazr.authentication==0.1.3,lazr.restfulclient==0.14.2,lazr.uri==1.0.3,libcharmstore==0.0.9,linecache2==1.0.0,macaroonbakery==1.2.3,MarkupSafe==1.1.1,mccabe==0.3.1,mock==3.0.5,monotonic==1.5,more-itertools==5.0.0,msgpack==0.6.2,munch==2.5.0,netaddr==0.7.19,netifaces==0.10.9,nose==1.3.7,oauth==1.0.1,oauthlib==3.1.0,openstacksdk==0.39.0,os-client-config==2.0.0,os-service-types==1.7.0,osc-lib==1.15.0,oslo.concurrency==3.31.0,oslo.config==7.0.0,oslo.context==2.23.0,oslo.i18n==3.25.1,oslo.log==3.45.2,oslo.serialization==2.29.2,oslo.utils==3.42.1,osprofiler==2.9.0,otherstuf==1.1.0,parse==1.14.0,path.py==11.5.2,pathlib2==2.3.5,pathspec==0.3.4,pbr==5.4.4,pep8==1.7.1,pika==0.13.1,pkg-resources==0.0.0,prettytable==0.7.2,protobuf==3.11.2,pycparser==2.19,pyflakes==0.8.1,pyinotify==0.9.6,pymacaroons==0.13.0,PyNaCl==1.3.0,pyOpenSSL==19.1.0,pyparsing==2.4.6,pyperclip==1.7.0,pyRFC3339==1.1,python-barbicanclient==4.9.0,python-ceilometerclient==2.9.0,python-cinderclient==4.3.0,python-dateutil==2.8.1,python-designateclient==3.0.0,python-glanceclient==2.17.0,python-heatclient==1.18.0,python-keystoneclient==3.22.0,python-manilaclient==1.29.0,python-mimeparse==1.6.0,python-neutronclient==6.14.0,python-novaclient==16.0.0,python-openstackclient==4.0.0,python-subunit==1.3.0,python-swiftclient==3.8.1,pytz==2019.3,pyudev==0.21.0,PyYAML==3.13,requests==2.22.0,requestsexceptions==1.4.0,rfc3986==1.3.2,ruamel.ordereddict==0.4.14,ruamel.yaml==0.15.100,scandir==1.10.0,SecretStorage==2.3.1,simplejson==3.17.0,six==1.13.0,stestr==2.6.0,stevedore==1.31.0,stuf==0.9.16,subprocess32==3.5.4,Tempita==0.5.2,testresources==2.0.1,testtools==2.3.0,theblues==0.5.2,traceback2==1.4.0,translationstring==1.3,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.25.7,vergit==1.0.2,virtualenv==16.7.9,voluptuous==0.11.7,wadllib==1.3.3,warlock==1.3.3,wcwidth==0.1.8,WebOb==1.8.5,websocket-client==0.40.0,wrapt==1.11.2,wsgi-intercept==1.9.1,zipp==0.6.0,zope.interface==4.7.1
func-smoke run-test-pre: PYTHONHASHSEED='0'
func-smoke runtests: commands[0] | bundletester -vl DEBUG -r json -o func-results.json gate-basic-xenial-queens --no-destroy
DEBUG:bundletester.utils:Updating JUJU_MODEL: "" -> "stsstack-stsstack:admin/lp1850634"
DEBUG:root:Bootstrap environment: stsstack-stsstack:admin/lp1850634
DEBUG:deployer.env:Connecting to stsstack-stsstack:admin/lp1850634...
DEBUG:jujuclient.connector:Connecting to wss://10.5.0.14:17070/model/5758a1f7-8fb1-42a8-8df9-d19c6bec7804/api
DEBUG:deployer.env:Connected.
DEBUG:deployer.env: Terminating machines forcefully
INFO:deployer.env: Waiting for machine termination
DEBUG:jujuclient.connector:Connecting to wss://10.5.0.14:17070/model/5758a1f7-8fb1-42a8-8df9-d19c6bec7804/api
DEBUG:root:Waiting for applications to be removed...
DEBUG:runner:call ['/home/freyes/Projects/charms/openstack/builds/keystone-ldap/.tox/func-smoke/bin/charm-proof'] (cwd: /tmp/bundletester-V3u4BE/keystone-ldap)
DEBUG:runner:I: `display-name` not provided, add for custom naming in the UI
DEBUG:runner:I: config.yaml: option ssl_key has no default value
DEBUG:runner:I: config.yaml: option ssl_cert has no default value
DEBUG:runner:I: config.yaml: option ldap-user has no default value
DEBUG:runner:I: config.yaml: option ldap-server has no default value
DEBUG:runner:I: config.yaml: option ssl_ca has no default value
DEBUG:runner:I: config.yaml: option ldap-password has no default value
DEBUG:runner:I: config.yaml: option domain-name has no default value
DEBUG:runner:I: config.yaml: option ldap-suffix has no default value
DEBUG:runner:I: config.yaml: option ldap-config-flags has no default value
DEBUG:runner:I: config.yaml: option tls-ca-ldap has no default value
DEBUG:runner:Exit Code: 0
DEBUG:deployer.env: Terminating machines forcefully
INFO:deployer.env: Waiting for machine termination
DEBUG:jujuclient.connector:Connecting to wss://10.5.0.14:17070/model/5758a1f7-8fb1-42a8-8df9-d19c6bec7804/api
DEBUG:root:Waiting for applications to be removed...
DEBUG:runner:call ['/tmp/bundletester-V3u4BE/keystone-ldap/tests/gate-basic-xenial-queens'] (cwd: /tmp/bundletester-V3u4BE/keystone-ldap)
DEBUG:runner:2020-01-13 18:41:24,080 __init__ INFO: OpenStackAmuletDeployment: init
DEBUG:runner:2020-01-13 18:41:24,080 _add_services INFO: OpenStackAmuletDeployment: adding services
DEBUG:runner:2020-01-13 18:41:24,080 _determine_branch_locations INFO: OpenStackAmuletDeployment: determine branch locations
DEBUG:runner:2020-01-13 18:41:27 Starting deployment of stsstack-stsstack:admin/lp1850634
DEBUG:runner:2020-01-13 18:41:29 Deploying applications...
DEBUG:runner:2020-01-13 18:41:29 Deploying application keystone using cs:~openstack-charmers-next/keystone-473
DEBUG:runner:2020-01-13 18:41:39 Deploying application keystone-ldap using /tmp/charmyxiNiN/xenial/keystone-ldap
DEBUG:runner:2020-01-13 18:44:09 Deploying application ldap-server using /tmp/charmokoKnX/xenial/charm-ldap-test-fixture
DEBUG:runner:2020-01-13 18:44:15 Deploying application percona-cluster using cs:~openstack-charmers-next/percona-cluster-358
DEBUG:runner:2020-01-13 18:44:28 Config specifies num units for subordinate: keystone-ldap
DEBUG:runner:2020-01-13 18:51:25 Adding relations...
DEBUG:runner:2020-01-13 18:51:25 Adding relation keystone:shared-db <-> percona-cluster:shared-db
DEBUG:runner:2020-01-13 18:51:26 Adding relation keystone:domain-backend <-> keystone-ldap:domain-backend
DEBUG:runner:2020-01-13 18:54:26 Deployment complete in 779.30 seconds
DEBUG:runner:2020-01-13 18:55:13,946 _configure_services INFO: OpenStackAmuletDeployment: configure services
DEBUG:runner:2020-01-13 18:55:17,951 __init__ INFO: Waiting on extended status checks...
DEBUG:runner:2020-01-13 18:55:17,952 _auto_wait_for_status INFO: Waiting for extended status on units for 5400s...
DEBUG:runner:2020-01-13 18:55:17,952 _auto_wait_for_status DEBUG: Default extended status wait match: contains READY (case-insensitive)
DEBUG:runner:2020-01-13 18:55:17,953 _auto_wait_for_status DEBUG: Excluding services from extended status match: ['mysql', 'mongodb']
DEBUG:runner:2020-01-13 18:55:17,954 _auto_wait_for_status DEBUG: Waiting up to 5400s for extended status on services: ['keystone-ldap', 'keystone', 'ldap-server', 'percona-cluster']
DEBUG:runner:2020-01-13 18:55:27,996 _auto_wait_for_status INFO: OK
DEBUG:runner:2020-01-13 18:56:33,309 get_default_keystone_session DEBUG: Authenticating keystone admin...
DEBUG:runner:Exit Code: 0
DEBUG:bundletester.utils:Updating JUJU_MODEL: "stsstack-stsstack:admin/lp1850634" -> ""
___________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
func-smoke: commands succeeded
congratulations :)
real 15m25,243s
user 0m9,443s
sys 0m4,199s
~ $ juju ssh keystone/0 sudo su -
root at juju-ec7804-lp1850634-0:~# vim /etc/keystone/domains/keystone.userdomain.conf
root at juju-ec7804-lp1850634-0:~# systemctl restart keystone
Failed to restart keystone.service: Unit keystone.service is masked.
root at juju-ec7804-lp1850634-0:~# systemctl restart apache2
root at juju-ec7804-lp1850634-0:~# cat /etc/keystone/domains/keystone.userdomain.conf
[ldap]
url = ldap://10.5.0.11
user = cn=admin,dc=test,dc=com
password = crapper
suffix = dc=test,dc=com
user_allow_create = False
user_allow_update = False
user_allow_delete = False
group_allow_create = False
group_allow_update = False
group_allow_delete = False
# Upstream release note for more context:
# Fixed the problem where Keystone indiscriminately return the first RDN
# as the user ID, regardless whether it matches the configured
# 'user_id_attribute' or not. This will break deployments where
# 'group_members_are_ids' are set to False and 'user_id_attribute' is not
# in the DN. This patch will perform a lookup by DN if the first RND does
# not match the configured 'user_id_attribute'.
###### Test scenario 1 (exercises else path in _dn_to_id) ######
# Prior to bug 1782922 fix, 'openstack user list --group cloud --domain userdomain'
# returns nothing.
# After bug 1782922 fix, 'openstack user list --group cloud --domain userdomain'
# returns users. _dn_to_id() takes new else path, where 'ID' attribute is not in
# the DN, and LDAP search is performed to look it up from the user entry itself.
group_id_attribute = businessCategory
group_name_attribute = businessCategory
group_member_attribute = member
group_members_are_ids = False
group_objectclass = groupOfNames
group_tree_dn = ou=groups,dc=test,dc=com
#user_id_attribute = uidNumber
user_objectclass = inetOrgPerson
user_tree_dn = ou=users,dc=test,dc=com
###### Test scenario 2 (exercises if path in _dn_to_id) ######
# Configuration is same as above except user_id_attribute not specified.
# User supplied configuration flags
[identity]
driver = ldap
root at juju-ec7804-lp1850634-0:~# add-apt-repository cloud-archive:queens-proposed
Ubuntu Cloud Archive for OpenStack Queens [proposed]
More info: https://wiki.ubuntu.com/OpenStack/CloudArchive
Press [ENTER] to continue or ctrl-c to cancel adding it
Reading package lists...
Building dependency tree...
Reading state information...
ubuntu-cloud-keyring is already the newest version (2012.08.14).
0 upgraded, 0 newly installed, 0 to remove and 12 not upgraded.
root at juju-ec7804-lp1850634-0:~# apt-get update -qq
0root at juju-ec7804-lp1850634-0:~# apt policy keystone
keystone:
Installed: 2:13.0.2-0ubuntu1~cloud0
Candidate: 2:13.0.2-0ubuntu3~cloud0
Version table:
2:13.0.2-0ubuntu3~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-proposed/queens/main amd64 Packages
*** 2:13.0.2-0ubuntu1~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/queens/main amd64 Packages
100 /var/lib/dpkg/status
2:9.3.0-0ubuntu3.2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
2:9.3.0-0ubuntu3.1 500
500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
2:9.0.0-0ubuntu1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
root at juju-ec7804-lp1850634-0:~# apt-get upgrade -y -qq
(Reading database ... 98483 files and directories currently installed.)
Preparing to unpack .../dh-python_3.20180325ubuntu2~cloud1_all.deb ...
Unpacking dh-python (3.20180325ubuntu2~cloud1) over (2.20151103ubuntu1.2) ...
Preparing to unpack .../libhogweed4_3.4-1~cloud0_amd64.deb ...
Unpacking libhogweed4:amd64 (3.4-1~cloud0) over (3.2-1ubuntu0.16.04.1) ...
Preparing to unpack .../libnettle6_3.4-1~cloud0_amd64.deb ...
Unpacking libnettle6:amd64 (3.4-1~cloud0) over (3.2-1ubuntu0.16.04.1) ...
Preparing to unpack .../dnsmasq-base_2.79-1~cloud0_amd64.deb ...
Unpacking dnsmasq-base (2.79-1~cloud0) over (2.75-1ubuntu0.16.04.5) ...
Preparing to unpack .../libnuma1_2.0.11-2.1ubuntu0.1~cloud0_amd64.deb ...
Unpacking libnuma1:amd64 (2.0.11-2.1ubuntu0.1~cloud0) over (2.0.11-1ubuntu1.1) ...
Preparing to unpack .../python3-cffi-backend_1.11.5-1~cloud0_amd64.deb ...
Unpacking python3-cffi-backend (1.11.5-1~cloud0) over (1.5.2-1ubuntu1) ...
Preparing to unpack .../python3-chardet_3.0.4-1~cloud0_all.deb ...
Unpacking python3-chardet (3.0.4-1~cloud0) over (2.3.0-2) ...
Preparing to unpack .../python3-dnspython_1.15.0-1~cloud0_all.deb ...
Unpacking python3-dnspython (1.15.0-1~cloud0) over (1.12.0-0ubuntu3) ...
Preparing to unpack .../python3-idna_2.6-1~cloud0_all.deb ...
Unpacking python3-idna (2.6-1~cloud0) over (2.0-3) ...
Preparing to unpack .../keystone_2%3a13.0.2-0ubuntu3~cloud0_all.deb ...
Unpacking keystone (2:13.0.2-0ubuntu3~cloud0) over (2:13.0.2-0ubuntu1~cloud0) ...
Preparing to unpack .../python-keystone_2%3a13.0.2-0ubuntu3~cloud0_all.deb ...
Unpacking python-keystone (2:13.0.2-0ubuntu3~cloud0) over (2:13.0.2-0ubuntu1~cloud0) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Processing triggers for dbus (1.10.6-1ubuntu3.5) ...
Setting up dh-python (3.20180325ubuntu2~cloud1) ...
Setting up libnettle6:amd64 (3.4-1~cloud0) ...
Setting up libhogweed4:amd64 (3.4-1~cloud0) ...
Setting up dnsmasq-base (2.79-1~cloud0) ...
Setting up libnuma1:amd64 (2.0.11-2.1ubuntu0.1~cloud0) ...
Setting up python3-cffi-backend (1.11.5-1~cloud0) ...
Setting up python3-chardet (3.0.4-1~cloud0) ...
Setting up python3-dnspython (1.15.0-1~cloud0) ...
Setting up python3-idna (2.6-1~cloud0) ...
Setting up python-keystone (2:13.0.2-0ubuntu3~cloud0) ...
Setting up keystone (2:13.0.2-0ubuntu3~cloud0) ...
apache2_invoke keystone.conf: no action - site was disabled by maintainer
Processing triggers for libc-bin (2.23-0ubuntu11) ...
root at juju-ec7804-lp1850634-0:~# apt policy keystone
keystone:
Installed: 2:13.0.2-0ubuntu3~cloud0
Candidate: 2:13.0.2-0ubuntu3~cloud0
Version table:
*** 2:13.0.2-0ubuntu3~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-proposed/queens/main amd64 Packages
100 /var/lib/dpkg/status
2:13.0.2-0ubuntu1~cloud0 500
500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/queens/main amd64 Packages
2:9.3.0-0ubuntu3.2 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
2:9.3.0-0ubuntu3.1 500
500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
2:9.0.0-0ubuntu1 500
500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
root at juju-ec7804-lp1850634-0:~# systemctl restart apache2
~ $ source ~/Projects/charms/openstack/openstack-charm-testing/novarcv3_project
~ $ openstack user list --domain userdomain
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 37a490a14a5201a644c695ed35a7e355e7eead80914a51cd406479214b20c357 | Jane Doe |
| 2a82add35294eef417cc36f1aa1d6eea00fb66553fff720b4751c4410577d91c | John Doe |
+------------------------------------------------------------------+----------+
~ $ openstack group list --domain userdomain
+------------------------------------------------------------------+-------+
| ID | Name |
+------------------------------------------------------------------+-------+
| c21e634e4bcb086cceed8c2bfdfdc4237729f6181d85ba52203f892d041b9ec8 | cloud |
+------------------------------------------------------------------+-------+
~ $ openstack user list --group cloud --domain userdomain
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 2a82add35294eef417cc36f1aa1d6eea00fb66553fff720b4751c4410577d91c | John Doe |
| 37a490a14a5201a644c695ed35a7e355e7eead80914a51cd406479214b20c357 | Jane Doe |
+------------------------------------------------------------------+----------+
** Tags removed: verification-queens-needed
** Tags added: verification-queens-done
** Tags removed: verification-needed
** Tags added: verification-done
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to keystone in Ubuntu.
https://bugs.launchpad.net/bugs/1850634
Title:
queens regresion: _dn_to_id() not using utf8_encode/decode
Status in Ubuntu Cloud Archive:
Invalid
Status in Ubuntu Cloud Archive queens series:
Fix Committed
Status in OpenStack Identity (keystone):
Triaged
Status in keystone package in Ubuntu:
Invalid
Status in keystone source package in Bionic:
Fix Committed
Bug description:
[Impact]
There's a regression in the LDAP common backend code due to a recent
stable/queens backport that shouldn't have been backported past
stable/rocky. It was backported as part of the fixes for
https://bugs.launchpad.net/bugs/1782922.
The following patch shouldn't have been backported to stable/queens:
https://review.opendev.org/#/c/672519/
The reason why is because the following patch, which switched to bytes_mode=False, doesn't exist in stable/queens:
https://review.opendev.org/#/c/613648/
In particular see the changes to _dn_to_id() in https://review.opendev.org/#/c/613648/4/keystone/identity/backends/ldap/common.py.
Those changes didn't happen in stable/queens so _dn_to_id should still
be UTF-8 encoding/decoding the appropriate fields. In other words it
should still be using the following in stable/queens:
if self.id_attr == utf8_decode(
ldap.dn.str2dn(utf8_encode(dn))[0][0][0].lower()):
return utf8_decode(ldap.dn.str2dn(utf8_encode(dn))[0][0][1])
[Test Case]
See test case in https://bugs.launchpad.net/bugs/1782922.
[Regression Potential]
The code that will be fixed for this bug (ie. the code in the if statement) is being reverted to what it used to be prior to the bug fix for https://bugs.launchpad.net/bugs/1782922. Prior to 1782922, _dn_to_id() used to only consist of the code that is in the if statment, so the regression potential is very low. Code will be tested to minimize regression potential and patch has been submitted upstream.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1850634/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list