[Bug 1820789] [NEW] Removing a hypervisor doesn't delete it entirely
Xav Paice
xav.paice at canonical.com
Tue Mar 19 03:54:07 UTC 2019
Public bug reported:
When removing a host (because it got rebuilt, for example), we use:
openstack compute service delete <someid>
I expected that to remove the hostname cleanly from the database (or at
least mark it as deleted) so that the hostname can be re-used. This
isn't the case, the host remained in the nova_api database in the
resource_providers table, and therefore could not be re-used.
Starting nova-compute on the host in this state resulted in:
2019-03-18 22:48:26.023 62597 ERROR nova.scheduler.client.report [req-445f587d-74e5-4a96-a5b5-4717f9095fb6 - - - - -] [req-1f1e781e-2ed8-4f6a-ac9e-93ecc450cec5] Failed to create resource provider record in placement API for UUID c6c4d923-1d0c-4f12-8505-d5af60c28ade. Got 409: {"errors": [{"status": 409, "request_id": "req-1f1e781e-2ed8-4f6a-ac9e-93ecc450cec5", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: host22.maas already exists. ", "title": "Conflict"}]}.
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager [req-445f587d-74e5-4a96-a5b5-4717f9095fb6 - - - - -] Error updating resources for node host22.maas.: ResourceProviderCreationFailed: Failed to create resource provider host22.maas
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager Traceback (most recent call last):
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7345, in update_available_resource_for_node
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 277, in inner
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return f(*args, **kwargs)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 572, in _init_compute_node
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._update(context, cn)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager inv_data,
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 1104, in set_inventory_for_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 665, in _ensure_resource_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 64, in wrapper
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return f(self, *a, **k)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 612, in _create_resource_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider host22.maas
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager
I was unable to clear the database entry:
mysql> delete from resource_providers where name='host22.maas';
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`nova_api`.`resource_providers`, CONSTRAINT `resource_providers_ibfk_1` FOREIGN KEY (`root_provider_id`) REFERENCES `resource_providers` (`id`))
I did, in the end, rename the host info in the table which allowed me to continue:
update resource_providers set name='host22.maas.deleted' where id='94';
** Affects: nova (Ubuntu)
Importance: Undecided
Status: New
** Tags: canonical-bootstack
** Also affects: nova (Ubuntu)
Importance: Undecided
Status: New
** No longer affects: charm-nova-cloud-controller
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1820789
Title:
Removing a hypervisor doesn't delete it entirely
Status in nova package in Ubuntu:
New
Bug description:
When removing a host (because it got rebuilt, for example), we use:
openstack compute service delete <someid>
I expected that to remove the hostname cleanly from the database (or
at least mark it as deleted) so that the hostname can be re-used.
This isn't the case, the host remained in the nova_api database in the
resource_providers table, and therefore could not be re-used.
Starting nova-compute on the host in this state resulted in:
2019-03-18 22:48:26.023 62597 ERROR nova.scheduler.client.report [req-445f587d-74e5-4a96-a5b5-4717f9095fb6 - - - - -] [req-1f1e781e-2ed8-4f6a-ac9e-93ecc450cec5] Failed to create resource provider record in placement API for UUID c6c4d923-1d0c-4f12-8505-d5af60c28ade. Got 409: {"errors": [{"status": 409, "request_id": "req-1f1e781e-2ed8-4f6a-ac9e-93ecc450cec5", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: host22.maas already exists. ", "title": "Conflict"}]}.
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager [req-445f587d-74e5-4a96-a5b5-4717f9095fb6 - - - - -] Error updating resources for node host22.maas.: ResourceProviderCreationFailed: Failed to create resource provider host22.maas
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager Traceback (most recent call last):
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7345, in update_available_resource_for_node
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 277, in inner
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return f(*args, **kwargs)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 572, in _init_compute_node
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._update(context, cn)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager inv_data,
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 1104, in set_inventory_for_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 665, in _ensure_resource_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 64, in wrapper
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return f(self, *a, **k)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 612, in _create_resource_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider host22.maas
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager
I was unable to clear the database entry:
mysql> delete from resource_providers where name='host22.maas';
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`nova_api`.`resource_providers`, CONSTRAINT `resource_providers_ibfk_1` FOREIGN KEY (`root_provider_id`) REFERENCES `resource_providers` (`id`))
I did, in the end, rename the host info in the table which allowed me to continue:
update resource_providers set name='host22.maas.deleted' where id='94';
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1820789/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list