[Bug 1588462] Re: Status set race condition in OpenStack charms with optional relations

[Alex Kavanagh]

I guess we're trying to minimise the changes to the full set of charms,
but I think that this might not be avoidable.

The main issue, as David has investigated, is that the optional
charm_func= function passed to assess_status call
'set_os_workload_status()' which sets the status, followed by
'assess_status()' doing the same thing after calling the charm_func.

The reason the optional charm_func sets the status is to discover what
it will be prior to returning it to assess_status() which then sets it.

Obviously, this is a design failure.  In charm_helpers, the consequence
of the optional relations charm_func function, is that
_determine_os_workload_status() is called TWICE, once before the
optional relations, and then afterwards as part of the optional
relations.

The fix is to change optional_relations such that it returns a
dictionary of the possible optional interfaces that can be dynamically
added to the required_interfaces, which is then passed to
assess_status() so that _determine_os_workload_status() [in
charmhelpers] is only called once.  I'll do this on glance and propose
it for review.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1588462

Title:
  Status set race condition in OpenStack charms with optional relations

Status in ceph-radosgw package in Juju Charms Collection:
  New
Status in cinder package in Juju Charms Collection:
  New
Status in glance package in Juju Charms Collection:
  New
Status in keystone package in Juju Charms Collection:
  New
Status in neutron-api package in Juju Charms Collection:
  New
Status in neutron-gateway package in Juju Charms Collection:
  New
Status in nova-cloud-controller package in Juju Charms Collection:
  New

Bug description:
  In charms that call assess_status() on every hook execution and that have
  optional relations there are two places where status_set() is called in
  set_os_workload_status() and make_assess_status_func()'s _assess_status_func()
  leading to a race condition in which a status may be overwritten.

  For charms like nova-cloud-controller where strangely we call
  set_os_workload_status() on each hook execution instead of assess_status() we
  only see this during pause and resume actions which do call assess_status().

  Regardless of __if__ you hit the race condition you can see the problem simply
  using print statements and an update-status hook run:
  'STATUS SET IN set_os_workload_status', 'active', 'Unit is ready'
  'STATUS SET IN _assess_status_func()', 'active', 'Unit is ready'

  If the first happened to be different the second would clobber it.

  In the set_os_workload_status() stack a comparison is made against the current
  status before setting status. The assess_status() stack needs to do the same
  possibly by calling set_os_workload_status() itself.

  The goal should be to have a single canonical source for status_set.

  This will affect all OpenStack charms that have optional relations.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charms/+source/ceph-radosgw/+bug/1588462/+subscriptions