[Bug 1158563] [NEW] After grizzly upgrade, EC2 API requests fail:Could not find: credential

[Adam Gandelman]

Public bug reported:

Upgrading a fairly standard Folsom setup to Grizzly using our packages.
After the updated config files are put in place and database has been
migrated, nova's EC2 API fails to authenticate requests with keystone.
The OSAPI end point works just fine.

On the client:

$ euca-describe-instances 
Unauthorized: Failure communicating with keystone

On the keystone server, with debug enabled:

2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ********************
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x2d64250 200 OK>}
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] REQUEST_METHOD = POST
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] PATH_INFO = /ec2tokens
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] CONTENT_LENGTH = 436
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] eventlet.posthooks = []
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] RAW_PATH_INFO = /v2.0/ec2tokens
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] REMOTE_ADDR = 192.168.20.50
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x2d56050>
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] wsgi.url_scheme = http
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] webob._body_file = (<LimitedLengthFile(<eventlet.wsgi.Input object at 0x2d56050>, maxlen=436)>, <eventlet.wsgi.Input object at 0x2d56050>)
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] SERVER_PORT = 5000
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] wsgi.input = <_io.BytesIO object at 0x2d5c1d0>
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] openstack.context = {'token_id': None, 'is_admin': False}
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] HTTP_HOST = test-09.os.magners.qa.lexington:5000
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] wsgi.multithread = True
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] openstack.params = {u'ec2Credentials': {u'access': u'8e283c4e394247fbbabe5474cdbda9e4', u'host': u'test-02.os.magners.qa.lexington:8773', u'verb': u'POST', u'params': {u'SignatureVersion': u'2', u'AWSAccessKeyId': u'8e283c4e394247fbbabe5474cdbda9e4', u'Timestamp': u'2013-03-21T22:53:36Z', u'SignatureMethod': u'HmacSHA256', u'Version': u'2010-08-31', u'Action': u'DescribeInstances'}, u'signature': u'gEJ42tacqfUrqvgpj2ouqg3T+aAiwzu6c5LWMrRQDEA=', u'path': u'/services/Cloud/'}}
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] wsgi.version = (1, 0)
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] SERVER_NAME = 192.168.20.57
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] wsgi.run_once = False
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x7f0bbbf08270>
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] wsgi.multiprocess = False
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] webob.is_body_seekable = True
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] CONTENT_TYPE = application/json
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = identity
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] 
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] {"ec2Credentials": {"access": "8e283c4e394247fbbabe5474cdbda9e4", "host": "test-02.os.magners.qa.lexington:8773", "verb": "POST", "params": {"SignatureVersion": "2", "AWSAccessKeyId": "8e283c4e394247fbbabe5474cdbda9e4", "Timestamp": "2013-03-21T22:53:36Z", "SignatureMethod": "HmacSHA256", "Version": "2010-08-31", "Action": "DescribeInstances"}, "signature": "gEJ42tacqfUrqvgpj2ouqg3T+aAiwzu6c5LWMrRQDEA=", "path": "/services/Cloud/"}}
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] 
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] arg_dict: {}
2013-03-21 18:53:36  WARNING [keystone.common.wsgi] Could not find: credential-8e283c4e394247fbbabe5474cdbda9e4
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] Content-Type = application/json
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] Content-Length = 120
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] 
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
2013-03-21 18:53:36    DEBUG [keystone.common.wsgi] {"error": {"message": "Could not find: credential-8e283c4e394247fbbabe5474cdbda9e4", "code": 404, "title": "Not Found"}}
2013-03-21 18:53:36     INFO [access] 192.168.20.50 - - [21/Mar/2013:22:53:36 +0000] "POST http://test-09.os.magners.qa.lexington:5000/v2.0/ec2tokens HTTP/1.0" 404 120
2013-03-21 18:53:36    DEBUG [eventlet.wsgi.server] 192.168.20.50 - - [21/Mar/2013 18:53:36] "POST /v2.0/ec2tokens HTTP/1.1" 404 256 0.010670

The nova-api-ec2.log:

2013-03-21 18:54:28.516 13600 ERROR nova.api.ec2 [-] Unauthorized: Failure communicating with keystone
2013-03-21 18:54:28.516 13600 INFO nova.api.ec2 [-] 0.14199s 192.168.20.1 POST /services/Cloud/ None:None 400 [Boto/2.3.0 (linux2)] application/x-www-form-urlencoded text/xml
2013-03-21 18:54:28.517 13600 INFO nova.ec2.wsgi.server [-] 192.168.20.1 "POST /services/Cloud/ HTTP/1.1" status: 400 len: 327 time: 0.0149109

** Affects: keystone
     Importance: Undecided
         Status: New

** Affects: nova
     Importance: Undecided
         Status: New

** Affects: keystone (Ubuntu)
     Importance: Undecided
         Status: New

** Affects: nova (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: upgrade

** Also affects: nova
   Importance: Undecided
       Status: New

** Also affects: keystone (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: keystone
   Importance: Undecided
       Status: New

** Tags added: upgrade

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1158563

Title:
  After grizzly upgrade, EC2 API requests fail:Could not find:
  credential

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1158563/+subscriptions