[Bug 1904850] [NEW] Gnocchi plugin fails to load when there is issue with Keystone connection

Gautam Prasad 1904850 at bugs.launchpad.net
Thu Nov 19 09:24:00 UTC 2020 

Public bug reported:

When openstack services are restarted and Keystone is not fully up or is
up but failed to respond the connection request, then Gnocchi publisher
plugin fails to load.

I am not sure if the Gnocchi publisher can be refactored to allow plugin
to load initially and initialize/retry for keystone connection later
when it becomes available.. ?

Openstack release: ussuri.


2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base [-] Unable to load publisher gnocchi://: keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://<host>:5000/v3/auth/tokens timed out
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     six.raise_from(e, None)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "<string>", line 3, in raise_from
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     httplib_response = conn.getresponse()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     response.begin()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib64/python3.6/http/client.py", line 307, in begin
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     version, status, reason = self._read_status()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib64/python3.6/socket.py", line 586, in readinto
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return self._sock.recv_into(b)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 328, in recv_into
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return self.recv_into(*args, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 326, in recv_into
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     raise timeout("The read operation timed out")
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base socket.timeout: The read operation timed out
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base During handling of the above exception, another exception occurred:
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     timeout=timeout
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     raise six.reraise(type(error), error, _stacktrace)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 703, in reraise


....


2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/pipeline/base.py", line 124, in __init__
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     self.publishers.append(publisher_manager.get(p))
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/pipeline/base.py", line 179, in get
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     'ceilometer.%s.publisher' % self._purpose)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/publisher/__init__.py", line 36, in get_publisher
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return loaded_driver.driver(conf, parse_result)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/publisher/gnocchi.py", line 249, in __init__
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     conf, request_timeout=timeout)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/gnocchi_client.py", line 39, in get_gnocchiclient
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     region_name=region_name)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1225, in get_endpoint
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return auth.get_endpoint(self, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     allow_version_hack=allow_version_hack, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     service_catalog = self.get_access(session).service_catalog
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     self.auth_ref = self.get_auth_ref(session)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     authenticated=False, log=False, **rkwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1131, in post
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return self.request(url, 'POST', **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     resp = send(**kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in _send_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     raise exceptions.ConnectTimeout(msg)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://<host>:5000/v3/auth/tokens timed out

** Affects: ceilometer (Ubuntu)
     Importance: Undecided
         Status: New

** Summary changed:

- Gnocchi plugin fails to load when there is issue with Keystrone connection
+ Gnocchi plugin fails to load when there is issue with Keystone connection

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceilometer in Ubuntu.
https://bugs.launchpad.net/bugs/1904850

Title:
  Gnocchi plugin fails to load when there is issue with Keystone
  connection

Status in ceilometer package in Ubuntu:
  New

Bug description:
  When openstack services are restarted and Keystone is not fully up or
  is up but failed to respond the connection request, then Gnocchi
  publisher plugin fails to load.

  I am not sure if the Gnocchi publisher can be refactored to allow
  plugin to load initially and initialize/retry for keystone connection
  later when it becomes available.. ?

  Openstack release: ussuri.

  
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base [-] Unable to load publisher gnocchi://: keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://<host>:5000/v3/auth/tokens timed out
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     six.raise_from(e, None)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "<string>", line 3, in raise_from
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     httplib_response = conn.getresponse()
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     response.begin()
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib64/python3.6/http/client.py", line 307, in begin
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     version, status, reason = self._read_status()
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib64/python3.6/socket.py", line 586, in readinto
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return self._sock.recv_into(b)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 328, in recv_into
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return self.recv_into(*args, **kwargs)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 326, in recv_into
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     raise timeout("The read operation timed out")
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base socket.timeout: The read operation timed out
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base During handling of the above exception, another exception occurred:
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     timeout=timeout
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     raise six.reraise(type(error), error, _stacktrace)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 703, in reraise

  
  ....

  
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/pipeline/base.py", line 124, in __init__
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     self.publishers.append(publisher_manager.get(p))
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/pipeline/base.py", line 179, in get
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     'ceilometer.%s.publisher' % self._purpose)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/publisher/__init__.py", line 36, in get_publisher
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return loaded_driver.driver(conf, parse_result)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/publisher/gnocchi.py", line 249, in __init__
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     conf, request_timeout=timeout)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/ceilometer/gnocchi_client.py", line 39, in get_gnocchiclient
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     region_name=region_name)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1225, in get_endpoint
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return auth.get_endpoint(self, **kwargs)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     allow_version_hack=allow_version_hack, **kwargs)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     service_catalog = self.get_access(session).service_catalog
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     self.auth_ref = self.get_auth_ref(session)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     authenticated=False, log=False, **rkwargs)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1131, in post
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     return self.request(url, 'POST', **kwargs)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     resp = send(**kwargs)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base   File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in _send_request
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base     raise exceptions.ConnectTimeout(msg)
  2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://<host>:5000/v3/auth/tokens timed out

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceilometer/+bug/1904850/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list