[Bug 1588462] [NEW] Status set race condition in OpenStack charms with optional relations

[David Ames]

Public bug reported:

In charms that call assess_status() on every hook execution and that have
optional relations there are two places where status_set() is called in
set_os_workload_status() and make_assess_status_func()'s _assess_status_func()
leading to a race condition in which a status may be overwritten.

For charms like nova-cloud-controller where strangely we call
set_os_workload_status() on each hook execution instead of assess_status() we
only see this during pause and resume actions which do call assess_status().

Regardless of __if__ you hit the race condition you can see the problem simply
using print statements and an update-status hook run:
'STATUS SET IN set_os_workload_status', 'active', 'Unit is ready'
'STATUS SET IN _assess_status_func()', 'active', 'Unit is ready'

If the first happened to be different the second would clobber it.

In the set_os_workload_status() stack a comparison is made against the current
status before setting status. The assess_status() stack needs to do the same
possibly by calling set_os_workload_status() itself.

The goal should be to have a single canonical source for status_set.

This will affect all OpenStack charms that have optional relations.

** Affects: ceph-radosgw (Juju Charms Collection)
     Importance: Undecided
         Status: New

** Affects: cinder (Juju Charms Collection)
     Importance: Undecided
         Status: New

** Affects: glance (Juju Charms Collection)
     Importance: Undecided
         Status: New

** Affects: keystone (Juju Charms Collection)
     Importance: Undecided
         Status: New

** Affects: neutron-api (Juju Charms Collection)
     Importance: Undecided
         Status: New

** Affects: neutron-gateway (Juju Charms Collection)
     Importance: Undecided
         Status: New

** Affects: nova-cloud-controller (Juju Charms Collection)
     Importance: Undecided
         Status: New

** Also affects: nova-cloud-controller (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: neutron-api (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: cinder (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: glance (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: ceph-radosgw (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: neutron-gateway (Juju Charms Collection)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1588462

Title:
  Status set race condition in OpenStack charms with optional relations

Status in ceph-radosgw package in Juju Charms Collection:
  New
Status in cinder package in Juju Charms Collection:
  New
Status in glance package in Juju Charms Collection:
  New
Status in keystone package in Juju Charms Collection:
  New
Status in neutron-api package in Juju Charms Collection:
  New
Status in neutron-gateway package in Juju Charms Collection:
  New
Status in nova-cloud-controller package in Juju Charms Collection:
  New

Bug description:
  In charms that call assess_status() on every hook execution and that have
  optional relations there are two places where status_set() is called in
  set_os_workload_status() and make_assess_status_func()'s _assess_status_func()
  leading to a race condition in which a status may be overwritten.

  For charms like nova-cloud-controller where strangely we call
  set_os_workload_status() on each hook execution instead of assess_status() we
  only see this during pause and resume actions which do call assess_status().

  Regardless of __if__ you hit the race condition you can see the problem simply
  using print statements and an update-status hook run:
  'STATUS SET IN set_os_workload_status', 'active', 'Unit is ready'
  'STATUS SET IN _assess_status_func()', 'active', 'Unit is ready'

  If the first happened to be different the second would clobber it.

  In the set_os_workload_status() stack a comparison is made against the current
  status before setting status. The assess_status() stack needs to do the same
  possibly by calling set_os_workload_status() itself.

  The goal should be to have a single canonical source for status_set.

  This will affect all OpenStack charms that have optional relations.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charms/+source/ceph-radosgw/+bug/1588462/+subscriptions