[Bug 2158101] Re: Masakari delays BFV instance evacuation when Cinder volume has multiattach=False on shared storage backend

Wed Sep 2 12:41:34 UTC 2026

** Changed in: masakari (Ubuntu Stonking)
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2158101

Title:
  Masakari delays BFV instance evacuation when Cinder volume has
  multiattach=False on shared storage backend

Status in Ubuntu Cloud Archive caracal series:
  New
Status in Ubuntu Cloud Archive epoxy series:
  New
Status in Ubuntu Cloud Archive gazpacho series:
  New
Status in Ubuntu Cloud Archive hibiscus series:
  New
Status in masakari:
  Fix Committed
Status in OpenStack Snap:
  Incomplete
Status in masakari package in Ubuntu:
  Fix Released
Status in masakari source package in Noble:
  New
Status in masakari source package in Resolute:
  New
Status in masakari source package in Stonking:
  Fix Released

Bug description:
  ## Summary

  Masakari fails to evacuate boot-from-volume (BFV) instances promptly
  during host failure when the Cinder volume type has
  `multiattach=False`. The evacuation is delayed by ~13 minutes before
  being re-queued, and completes only after the source hypervisor
  recovers and clears the iSCSI session, or after manual Cinder volume
  state reset.

  ## Environment

  - **Deployment method:** OpenStack Sunbeam
  - **OpenStack release:** 2024.1 (Nova 29.2.0)
  - **Masakari:** 2024.1/stable (masakari-k8s charm rev 137)
  - **Cinder backend:** Pure Storage FA iSCSI (PureISCSIDriver)
  - **Volume multiattach:** False (default)
  - **Cluster:** 3-node, all nodes are control + compute + storage

  ## Steps to reproduce

  1. Create a BFV instance using a Cinder volume with `multiattach=False` on a Pure Storage iSCSI backend
  2. Configure Masakari with a failover segment covering the host
  3. Hard power-off the host (IPMI/BMC hard power cut)
  4. Observe Masakari's evacuation behavior

  ## Observed behavior

  Masakari's `EvacuateInstancesTask` logs `"Evacuation of instance
  started"` for all instances, including BFV ones, at T+0. However:

  - **Ephemeral instances (4/6):** evacuated successfully within 2min 26s
  - **BFV instances (2/6):** no Nova server event recorded for ~13 minutes after Masakari triggered evacuation. Nova migration records are only created at T+13min. Actual rebuild completed in under 1 minute once Nova accepted the request.

  Total time from host failure to BFV instance availability on
  destination: **~37 minutes** (13min delay + wait for source node
  recovery to clear iSCSI session when powering on the host).

  ## Evidence

  **Masakari notification detail (EvacuateInstancesTask):**

  ```
  19:24:20 - "Evacuation of instance started: 'caad8b3d-c2c3-43ec-845f-7c5f056ebcd4'"
  19:24:20 - "Evacuation of instance started: '4efcc1bf-8c3b-4467-8f75-420ff0b3863a'"
  19:26:47 - "Evacuation process completed!"
  ```

  **Nova server events (`openstack server event list --long`):**
  - No evacuation event recorded between 19:24:20 and 19:37:51 for either BFV VM
  - First Nova evacuation event: 19:37:51 (caad8b3d) and 19:37:59 (4efcc1bf)
  - compute_rebuild_instance completed: 19:38:21 and 19:38:55 (~1min after Nova accepted)

  **Volume state:**

  ```
  $ openstack volume show 35ab2ef8-5508-4401-bff4-06c2f119c225 | grep multiattach
  | multiattach | False |
  ```

  ## Note on log availability

  The Masakari, Nova conductor, Nova API, and Cinder API pod logs
  covering the critical 19:24–19:37 window were no longer available at
  the time of the investigation. The failed node was rebooted as part of
  the test recovery procedure, resulting in the loss of all pod logs on
  that node. The surviving nodes' pod logs had also rotated by the time
  the investigation was conducted (~2 days after the test).

  As a result, it was not possible to confirm from the logs whether:

  - Masakari submitted the initial evacuation request to Nova and received an error response (e.g., volume `in-use`, Nova rejecting the evacuate call), or
  - Masakari's internal evacuation loop detected the volume issue and deferred the request before reaching Nova

  The ~13-minute gap between Masakari's "Evacuation of instance started"
  log (19:24:20) and the first Nova server event (19:37:51) is
  consistent with Masakari's `FixedIntervalWithTimeoutLoopingCall`
  polling at `verify_interval=60s` - suggesting approximately 13 polling
  cycles elapsed before Masakari re-submitted the evacuation. The exact
  failure mode during the first attempt could not be confirmed due to
  log rotation.

  To reproduce and capture the full log sequence, the test should be
  repeated with extended log retention and pod log `--previous` capture
  immediately after the event.

  ## Root cause analysis

  Masakari's `EvacuateInstancesTask` calls Nova's evacuate API for all
  instances without first checking whether instances are boot-from-
  volume and whether their Cinder volumes need to be force-detached from
  the dead host.

  With `multiattach=False`, Nova cannot attach the volume to the
  destination host until it has detached it from the source. Since the
  source host is powered off, Nova cannot cleanly detach the iSCSI
  session - it requires either:

  1. The source hypervisor to come back online and clear the session, or
  2. A `os-force_detach` call to Cinder to forcibly clear the volume attachment

  Masakari currently does neither. It calls Nova's evacuate API and then
  polls for confirmation (`FixedIntervalWithTimeoutLoopingCall` with
  `verify_interval=60s`). When Nova cannot complete the evacuation
  (volume still `in-use`), Masakari eventually re-queues the evacuation
  after its internal retry logic (~13 minutes in our test), at which
  point it succeeds because enough time has passed for the volume state
  to change.

  ## Expected behavior

  Masakari should detect BFV instances and call `os-force_detach` on
  their Cinder volumes before or during the evacuation call to the Nova
  API. Since the backend is shared storage (Pure Storage FA, Ceph,
  etc.), the storage array can terminate the dead host's iSCSI/RBD
  session without requiring the hypervisor to be alive. This would allow
  Nova to complete the evacuation immediately, reducing BFV evacuation
  time from ~37 minutes to under 2 minutes (matching ephemeral
  instances).

  ## Workaround

  Manually reset Cinder volume state immediately after host power-off:

  ```bash
  openstack volume set --state available <volume-id>
  ```

  Or enable `multiattach` on the Pure Storage volume types, which allows
  Nova to attach to the destination before detaching from the source:

  ```bash
  openstack volume type set --property multiattach='<is> True' <volume-type>
  ```

  ## Impact

  Any Masakari deployment that uses BFV instances with
  `multiattach=False` on a shared storage backend (Pure Storage, NetApp,
  etc.) will experience significantly delayed evacuation in the event of
  a hard host failure. This makes Masakari HA effectively unusable for
  BFV workloads without the multiattach workaround.

  ## References

  - https://canonical-openstack.readthedocs-hosted.com/en/latest/how-
  to/features/instance-recovery/#supplementary-information

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/caracal/+bug/2158101/+subscriptions