[Bug 1534655] Re: Autoscaling auth failure in OpenStack Kilo 2015.1.2

melanie witt 1534655 at bugs.launchpad.net
Fri Apr 15 22:44:54 UTC 2016 

*** This bug is a duplicate of bug 1522402 ***
    https://bugs.launchpad.net/bugs/1522402

At present, the only way to make the novaclient Python API work with
keystone v3 is to construct a keystoneclient Session object and pass it
to novaclient Client when creating the client object.

>From comment #11 on bug 1522402:

"http://docs.openstack.org/developer/python-novaclient/api.html#usage

Note that you must create a Session object and pass it to
novaclient.Client(session=<session object>) when creating the novaclient
object. If you don't, novaclient will fall back on its own code for
keystone auth and will NOT work with keystone v3."

** This bug has been marked a duplicate of bug 1522402
   Novaclient does not support v3 Keystone API

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to heat in Ubuntu.
https://bugs.launchpad.net/bugs/1534655

Title:
  Autoscaling auth failure in OpenStack Kilo 2015.1.2

Status in heat:
  New
Status in OpenStack Identity (keystone):
  Incomplete
Status in python-novaclient:
  New
Status in heat package in Ubuntu:
  Confirmed

Bug description:
  We upgraded environment to OpenStack Kilo 2015.1.2 and then we cannot
  use heat autoscaling feature anymore.

  Keystone version 1:2015.1.2-0ubuntu2~cloud0
  Heat version 2015.1.2-0ubuntu1~cloud0

  - First we hit authentication failure on ec2tokens heat-api-cfn.log

  2016-01-15 14:21:07.505 6590 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
  2016-01-15 14:21:07.507 6590 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
  2016-01-15 14:21:07.509 6590 INFO heat.api.aws.ec2token [-] Authenticating with http://10.200.26.10:5000/ec2tokens
  2016-01-15 14:21:07.533 6590 INFO heat.api.aws.ec2token [-] AWS authentication failure.
  2016-01-15 14:21:07.535 6590 INFO eventlet.wsgi.server [-] 10.200.26.11 - - [15/Jan/2016 14:21:07] "POST /v1/signal/arn%3Aopenstack%3Aheat%3A%3A4a28da719dc64deb9d1d5cfa770bbca2%3Astacks%2Fheat_autoscale_cpu_vpc20%2F3a268b22-d436-4cf8-9088-8418702be09a%2Fresources%2Fserver_scaleup_policy?Timestamp=2016-01-15T13%3A14%3A22Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=f042045aabc74728a49059a4c52226bf&SignatureVersion=2&Signature=GomCM13riX1UYu3OpJZ3ERXpfekHkmeACA4DznTL6g8%3D HTTP/1.1" 403 301 0.031258

  We applied patch to keystone  https://bugs.launchpad.net/openstack-
  ansible/+bug/1515485
  https://git.openstack.org/cgit/openstack/keystone/commit/?id=f66badae63b1544c3bcc83e0c1afc8e9cdc71bab

  Then we hit error on python-novaclient auth boot during autoscaling
  heat-engine.log

  2016-01-15 06:43:05.836 11301 ERROR heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (NotFound(),)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack Traceback (most recent call last):
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     result = res.validate()
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     limits = self.client_plugin().absolute_limits()
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     limits = self.client().limits.get()
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     return self._get("/limits%s" % query_string, "limits")
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     _resp, body = self.api.client.get(url)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 440, in get
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     return self._cs_request(url, 'GET', **kwargs)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 427, in _cs_request
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     self.authenticate()
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 569, in authenticate
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     self._v2_auth(auth_url)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 634, in _v2_auth
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     return self._authenticate(url, body)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 647, in _authenticate
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     **kwargs)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 392, in _time_request
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     resp, body = self.request(url, method, **kwargs)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 386, in request
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack     raise exceptions.from_response(resp, body, url, method)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack NotFound: The resource could not be found. (HTTP 404)
  2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack
  2016-01-15 06:43:05.838 11301 INFO heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Failed to validate: The resource could not be found. (HTTP 404)

  It was caused by bad auth_url argument passed into novaclient -
  http://10.0.170.10:5000/v3/v2.0

  We modified /usr/lib/python2.7/dist-
  packages/heat/engine/clients/os/nova.py to replace v3 to v2.0

          args = {
              'project_id': self.context.tenant,
              'auth_url': self.context.auth_url.replace('/v3','/v2.0'),
              'service_type': 'compute',
              'username': None,
              'api_key': None,
              'extensions': extensions,
              'endpoint_type': endpoint_type,
              'http_log_debug': self._get_client_option('nova',
                                                        'http_log_debug'),
              'cacert': self._get_client_option('nova', 'ca_file'),
              'insecure': self._get_client_option('nova', 'insecure')
          }

  And then we got following error heat-engine.log

  2016-01-15 14:36:05.464 8195 ERROR heat.engine.stack [req-b19963cf-81d3-482d-b31a-27d7b0862b89 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (ClientException(),)
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack Traceback (most recent call last):
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     result = res.validate()
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     limits = self.client_plugin().absolute_limits()
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     limits = self.client().limits.get()
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     return self._get("/limits%s" % query_string, "limits")
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     _resp, body = self.api.client.get(url)
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in get
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     return self._cs_request(url, 'GET', **kwargs)
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 426, in _cs_request
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     self.authenticate()
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 561, in authenticate
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     auth_url = self._v1_auth(auth_url)
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 609, in _v1_auth
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack     raise exceptions.from_response(resp, body, url)
  2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack ClientException: Unknown Error (HTTP 300)

  And then we have found that there is only auth_token and no username and password. It seems that python-novaclient cannot use v3 authentication token over v2.0.
   https://bugs.launchpad.net/python-novaclient/+bug/1522402

  Has anybody tested heat autoscaling on latest Kilo update?

To manage notifications about this bug go to:
https://bugs.launchpad.net/heat/+bug/1534655/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list