[Bug 1719770] Re: hypervisor stats issue after charm removal if nova-compute service not disabled first

[Edward Hope-Morley]

Hi @afreiberger, i've run a test using our charms and Xenial Mitaka and
here are the results - http://pastebin.ubuntu.com/25879833/

A few points for clarification:

When you do juju remove-unit nova-compute/0 the charm is not performing
any cleanup at all i.e. it will not notify the cloud controller that the
node is to be disabled or deleted. So what you are left with following a
remove-unit is a compute node whose database state represents it as
"State:down" "disabled:0" and "deleted:0". Therefore Nova is correctly
still counting the compute resources associated with that node as
available.

If, subsequent to removing the unit, you then issue a 'openstack compute
service delete <id>' the node is marked as "deleted:<id>" and it's
resources are no longer counted as available (as can be clearly seen in
the pastebin output I provided above).

I am still verifying whether entries with the same hostname but
different deleted status are skewing the stats and will report back once
ive confirmed but in any case the findings so far hopefully show that
regardless you always need to manually delete a compute host using the
api following its removal with juju.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1719770

Title:
  hypervisor stats issue after charm removal if nova-compute service not
  disabled first

Status in OpenStack nova-compute charm:
  Invalid
Status in nova package in Ubuntu:
  Incomplete

Bug description:
  In an environment with 592 physical threads (lscpu |grep '^CPU.s' and
  openstack hypervisor show -f value -c vcpus both show correct counts)
  I am seeing 712 vcpus. (likely also seeing inflated memory_mb and
  other stats due to the issue.)

  Querying the nova services DB table, I see:
  http://pastebin.ubuntu.com/25624553/

  It appears that of the 6 machines showing deleted in the services
  table, only one is showing as disabled.

  Digging through the nova/db/sqlalchemy/api.py code, it appears that
  there are filters on the hypervisor stats for Service.disabled ==
  false() and Service.binary == 'nova-compute', but I don't see it
  filtering for deleted == 0.

  I'm not exactly certain of the timeline of my uninstall and reinstall
  of the nova-compute units on the 6 x 24vcpu servers happened (see
  *-ST-{1,2} nova-compute services) that caused this behavior of the
  services not getting disabled, but nova api for hypervisor stats might
  be well served to filter out deleted services as well as disabled
  services, or if a deleted service should never not be disabled, nova
  service-delete should also set the disabled flag for the service.

  These services and compute_nodes do not show up in openstack
  hypervisor list.

  Site is running up-to-date Xenial/Mitaka on openstack-charmers 17.02.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-compute/+bug/1719770/+subscriptions