[Bug 2158101] Re: [SRU] Masakari marks host recovery finished with pending VM moves after Nova or Keystone API failure

Mon Sep 7 19:56:19 UTC 2026

** Merge proposal linked:
   https://code.launchpad.net/~gboutry/ubuntu/+source/masakari/+git/masakari/+merge/510946

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2158101

Title:
  [SRU] Masakari marks host recovery finished with pending VM moves
  after Nova or Keystone API failure

Status in Ubuntu Cloud Archive caracal series:
  New
Status in Ubuntu Cloud Archive epoxy series:
  New
Status in Ubuntu Cloud Archive gazpacho series:
  New
Status in Ubuntu Cloud Archive hibiscus series:
  Fix Released
Status in masakari:
  Fix Committed
Status in OpenStack Snap:
  Incomplete
Status in masakari package in Ubuntu:
  Fix Released
Status in masakari source package in Noble:
  New
Status in masakari source package in Resolute:
  In Progress
Status in masakari source package in Stonking:
  Fix Released

Bug description:
  [ Impact ]

  When Masakari handles a compute-host failure, a transient failure while
  querying Nova or Keystone can escape from an evacuation worker before the corresponding
  VM move is updated.

  The VM move remains in PENDING state, but Masakari can mark the overall
  host-failure notification as finished. Operators are therefore told that
  recovery completed even though an instance was not evacuated and may remain
  unavailable.

  The update ensures that these failures mark the VM move and host-recovery
  notification as failed instead of silently reporting successful recovery.

  [ Test Plan ]

  This test requires an OpenStack deployment with Masakari and at least two Nova
  compute hosts.

  1. Install the release version of Masakari from the archive.

  2. Create an ACTIVE instance on the first compute host and set its
     HA_Enabled property to True.

  3. While processing a compute-host failure notification, make the Nova
     server-detail request for that instance return HTTP 503.

  4. Confirm the existing bug:

     * the Masakari notification reaches FINISHED;
     * the instance has not been evacuated; and
     * the corresponding vmove remains PENDING without completion timestamps.

  5. Restore the compute host and Masakari host state.

  6. Install the Masakari packages from proposed and confirm the
     installed binary package version.

  7. Repeat steps 2 through 4 with the same injected Nova failure.

  8. Confirm the corrected behaviour:

     * the Masakari notification reaches FAILED;
     * the associated vmove records reach FAILED;
     * failed vmoves have an end time and error message; and
     * no vmove associated with the notification remains PENDING.

  9. Remove the injected Nova failure and perform another host evacuation with
     an ACTIVE HA-enabled instance.

  10. Confirm that the normal evacuation succeeds:

      * the notification reaches FINISHED;
      * the vmove reaches SUCCEEDED;
      * Nova reports the instance ACTIVE on the second compute;
      * the instance is running under libvirt on the second compute;
      * its Neutron port is ACTIVE and bound to the second compute; and
      * both compute services can be restored to the up/enabled state.

  [ Where problems could occur ]

  The change affects the compute-host recovery path, including instance lookup,
  instance locking and unlocking, and the final determination of whether host
  recovery succeeded.

  An error in this logic could incorrectly fail a successful evacuation, leave
  an instance locked, or allow a failed evacuation to be reported as successful.

  The added pending-vmove check could also expose an existing workflow or
  persistence race as a failed notification. The test plan therefore covers both
  the error path and a normal successful evacuation between two compute hosts.

  [ Other Info ]

  Upstream fix:

  https://opendev.org/openstack/masakari/commit/4e6dd2e645d71439eb289a1523a5d04c402e90bb

  The upstream source containing this fix uses a newer
  oslo.versionedobjects decorator API than the version available in Resolute/Noble.
  The backport retains Resolute/Noble's existing decorator API and includes only the
  behavioural changes required to fix this bug.

  --- previous description
  ## Summary

  Masakari fails to evacuate boot-from-volume (BFV) instances promptly
  during host failure when the Cinder volume type has
  `multiattach=False`. The evacuation is delayed by ~13 minutes before
  being re-queued, and completes only after the source hypervisor
  recovers and clears the iSCSI session, or after manual Cinder volume
  state reset.

  ## Environment

  - **Deployment method:** OpenStack Sunbeam
  - **OpenStack release:** 2024.1 (Nova 29.2.0)
  - **Masakari:** 2024.1/stable (masakari-k8s charm rev 137)
  - **Cinder backend:** Pure Storage FA iSCSI (PureISCSIDriver)
  - **Volume multiattach:** False (default)
  - **Cluster:** 3-node, all nodes are control + compute + storage

  ## Steps to reproduce

  1. Create a BFV instance using a Cinder volume with `multiattach=False` on a Pure Storage iSCSI backend
  2. Configure Masakari with a failover segment covering the host
  3. Hard power-off the host (IPMI/BMC hard power cut)
  4. Observe Masakari's evacuation behavior

  ## Observed behavior

  Masakari's `EvacuateInstancesTask` logs `"Evacuation of instance
  started"` for all instances, including BFV ones, at T+0. However:

  - **Ephemeral instances (4/6):** evacuated successfully within 2min 26s
  - **BFV instances (2/6):** no Nova server event recorded for ~13 minutes after Masakari triggered evacuation. Nova migration records are only created at T+13min. Actual rebuild completed in under 1 minute once Nova accepted the request.

  Total time from host failure to BFV instance availability on
  destination: **~37 minutes** (13min delay + wait for source node
  recovery to clear iSCSI session when powering on the host).

  ## Evidence

  **Masakari notification detail (EvacuateInstancesTask):**

  ```
  19:24:20 - "Evacuation of instance started: 'caad8b3d-c2c3-43ec-845f-7c5f056ebcd4'"
  19:24:20 - "Evacuation of instance started: '4efcc1bf-8c3b-4467-8f75-420ff0b3863a'"
  19:26:47 - "Evacuation process completed!"
  ```

  **Nova server events (`openstack server event list --long`):**
  - No evacuation event recorded between 19:24:20 and 19:37:51 for either BFV VM
  - First Nova evacuation event: 19:37:51 (caad8b3d) and 19:37:59 (4efcc1bf)
  - compute_rebuild_instance completed: 19:38:21 and 19:38:55 (~1min after Nova accepted)

  **Volume state:**

  ```
  $ openstack volume show 35ab2ef8-5508-4401-bff4-06c2f119c225 | grep multiattach
  | multiattach | False |
  ```

  ## Note on log availability

  The Masakari, Nova conductor, Nova API, and Cinder API pod logs
  covering the critical 19:24–19:37 window were no longer available at
  the time of the investigation. The failed node was rebooted as part of
  the test recovery procedure, resulting in the loss of all pod logs on
  that node. The surviving nodes' pod logs had also rotated by the time
  the investigation was conducted (~2 days after the test).

  As a result, it was not possible to confirm from the logs whether:

  - Masakari submitted the initial evacuation request to Nova and received an error response (e.g., volume `in-use`, Nova rejecting the evacuate call), or
  - Masakari's internal evacuation loop detected the volume issue and deferred the request before reaching Nova

  The ~13-minute gap between Masakari's "Evacuation of instance started"
  log (19:24:20) and the first Nova server event (19:37:51) is
  consistent with Masakari's `FixedIntervalWithTimeoutLoopingCall`
  polling at `verify_interval=60s` - suggesting approximately 13 polling
  cycles elapsed before Masakari re-submitted the evacuation. The exact
  failure mode during the first attempt could not be confirmed due to
  log rotation.

  To reproduce and capture the full log sequence, the test should be
  repeated with extended log retention and pod log `--previous` capture
  immediately after the event.

  ## Root cause analysis

  Masakari's `EvacuateInstancesTask` calls Nova's evacuate API for all
  instances without first checking whether instances are boot-from-
  volume and whether their Cinder volumes need to be force-detached from
  the dead host.

  With `multiattach=False`, Nova cannot attach the volume to the
  destination host until it has detached it from the source. Since the
  source host is powered off, Nova cannot cleanly detach the iSCSI
  session - it requires either:

  1. The source hypervisor to come back online and clear the session, or
  2. A `os-force_detach` call to Cinder to forcibly clear the volume attachment

  Masakari currently does neither. It calls Nova's evacuate API and then
  polls for confirmation (`FixedIntervalWithTimeoutLoopingCall` with
  `verify_interval=60s`). When Nova cannot complete the evacuation
  (volume still `in-use`), Masakari eventually re-queues the evacuation
  after its internal retry logic (~13 minutes in our test), at which
  point it succeeds because enough time has passed for the volume state
  to change.

  ## Expected behavior

  Masakari should detect BFV instances and call `os-force_detach` on
  their Cinder volumes before or during the evacuation call to the Nova
  API. Since the backend is shared storage (Pure Storage FA, Ceph,
  etc.), the storage array can terminate the dead host's iSCSI/RBD
  session without requiring the hypervisor to be alive. This would allow
  Nova to complete the evacuation immediately, reducing BFV evacuation
  time from ~37 minutes to under 2 minutes (matching ephemeral
  instances).

  ## Workaround

  Manually reset Cinder volume state immediately after host power-off:

  ```bash
  openstack volume set --state available <volume-id>
  ```

  Or enable `multiattach` on the Pure Storage volume types, which allows
  Nova to attach to the destination before detaching from the source:

  ```bash
  openstack volume type set --property multiattach='<is> True' <volume-type>
  ```

  ## Impact

  Any Masakari deployment that uses BFV instances with
  `multiattach=False` on a shared storage backend (Pure Storage, NetApp,
  etc.) will experience significantly delayed evacuation in the event of
  a hard host failure. This makes Masakari HA effectively unusable for
  BFV workloads without the multiattach workaround.

  ## References

  - https://canonical-openstack.readthedocs-hosted.com/en/latest/how-
  to/features/instance-recovery/#supplementary-information

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/caracal/+bug/2158101/+subscriptions