[Bug 2158101] Re: [SRU] Masakari marks host recovery finished with pending VM moves after Nova or Keystone API failure

[Guillaume Boutry]

** Description changed:

  [ Impact ]
  
  When Masakari handles a compute-host failure, a transient failure while
  querying Nova or Keystone can escape from an evacuation worker before the corresponding
  VM move is updated.
  
  The VM move remains in PENDING state, but Masakari can mark the overall
  host-failure notification as finished. Operators are therefore told that
  recovery completed even though an instance was not evacuated and may remain
  unavailable.
  
  The update ensures that these failures mark the VM move and host-recovery
  notification as failed instead of silently reporting successful recovery.
  
  [ Test Plan ]
  
  This test requires an OpenStack deployment with Masakari and at least two Nova
  compute hosts.
  
  1. Install the release version of Masakari from the Resolute archive.
  
  2. Create an ACTIVE instance on the first compute host and set its
-    HA_Enabled property to True.
+    HA_Enabled property to True.
  
  3. While processing a compute-host failure notification, make the Nova
-    server-detail request for that instance return HTTP 503.
+    server-detail request for that instance return HTTP 503.
  
  4. Confirm the existing bug:
  
-    * the Masakari notification reaches FINISHED;
-    * the instance has not been evacuated; and
-    * the corresponding vmove remains PENDING without completion timestamps.
+    * the Masakari notification reaches FINISHED;
+    * the instance has not been evacuated; and
+    * the corresponding vmove remains PENDING without completion timestamps.
  
  5. Restore the compute host and Masakari host state.
  
- 6. Install the Masakari packages from resolute-proposed and confirm the
-    installed binary package version.
+ 6. Install the Masakari packages from proposed and confirm the
+    installed binary package version.
  
  7. Repeat steps 2 through 4 with the same injected Nova failure.
  
  8. Confirm the corrected behaviour:
  
-    * the Masakari notification reaches FAILED;
-    * the associated vmove records reach FAILED;
-    * failed vmoves have an end time and error message; and
-    * no vmove associated with the notification remains PENDING.
+    * the Masakari notification reaches FAILED;
+    * the associated vmove records reach FAILED;
+    * failed vmoves have an end time and error message; and
+    * no vmove associated with the notification remains PENDING.
  
  9. Remove the injected Nova failure and perform another host evacuation with
-    an ACTIVE HA-enabled instance.
+    an ACTIVE HA-enabled instance.
  
  10. Confirm that the normal evacuation succeeds:
  
-     * the notification reaches FINISHED;
-     * the vmove reaches SUCCEEDED;
-     * Nova reports the instance ACTIVE on the second compute;
-     * the instance is running under libvirt on the second compute;
-     * its Neutron port is ACTIVE and bound to the second compute; and
-     * both compute services can be restored to the up/enabled state.
+     * the notification reaches FINISHED;
+     * the vmove reaches SUCCEEDED;
+     * Nova reports the instance ACTIVE on the second compute;
+     * the instance is running under libvirt on the second compute;
+     * its Neutron port is ACTIVE and bound to the second compute; and
+     * both compute services can be restored to the up/enabled state.
  
  [ Where problems could occur ]
  
  The change affects the compute-host recovery path, including instance lookup,
  instance locking and unlocking, and the final determination of whether host
  recovery succeeded.
  
  An error in this logic could incorrectly fail a successful evacuation, leave
  an instance locked, or allow a failed evacuation to be reported as successful.
  
  The added pending-vmove check could also expose an existing workflow or
  persistence race as a failed notification. The test plan therefore covers both
  the error path and a normal successful evacuation between two compute hosts.
  
  [ Other Info ]
  
  Upstream fix:
  
  https://opendev.org/openstack/masakari/commit/4e6dd2e645d71439eb289a1523a5d04c402e90bb
  
  The upstream source containing this fix uses a newer
- oslo.versionedobjects decorator API than the version available in Resolute.
+ oslo.versionedobjects decorator API than the version available in Resolute/Noble.
  The backport retains Resolute's existing decorator API and includes only the
  behavioural changes required to fix this bug.
  
  --- previous description
  ## Summary
  
  Masakari fails to evacuate boot-from-volume (BFV) instances promptly
  during host failure when the Cinder volume type has `multiattach=False`.
  The evacuation is delayed by ~13 minutes before being re-queued, and
  completes only after the source hypervisor recovers and clears the iSCSI
  session, or after manual Cinder volume state reset.
  
  ## Environment
  
  - **Deployment method:** OpenStack Sunbeam
  - **OpenStack release:** 2024.1 (Nova 29.2.0)
  - **Masakari:** 2024.1/stable (masakari-k8s charm rev 137)
  - **Cinder backend:** Pure Storage FA iSCSI (PureISCSIDriver)
  - **Volume multiattach:** False (default)
  - **Cluster:** 3-node, all nodes are control + compute + storage
  
  ## Steps to reproduce
  
  1. Create a BFV instance using a Cinder volume with `multiattach=False` on a Pure Storage iSCSI backend
  2. Configure Masakari with a failover segment covering the host
  3. Hard power-off the host (IPMI/BMC hard power cut)
  4. Observe Masakari's evacuation behavior
  
  ## Observed behavior
  
  Masakari's `EvacuateInstancesTask` logs `"Evacuation of instance
  started"` for all instances, including BFV ones, at T+0. However:
  
  - **Ephemeral instances (4/6):** evacuated successfully within 2min 26s
  - **BFV instances (2/6):** no Nova server event recorded for ~13 minutes after Masakari triggered evacuation. Nova migration records are only created at T+13min. Actual rebuild completed in under 1 minute once Nova accepted the request.
  
  Total time from host failure to BFV instance availability on
  destination: **~37 minutes** (13min delay + wait for source node
  recovery to clear iSCSI session when powering on the host).
  
  ## Evidence
  
  **Masakari notification detail (EvacuateInstancesTask):**
  
  ```
  19:24:20 - "Evacuation of instance started: 'caad8b3d-c2c3-43ec-845f-7c5f056ebcd4'"
  19:24:20 - "Evacuation of instance started: '4efcc1bf-8c3b-4467-8f75-420ff0b3863a'"
  19:26:47 - "Evacuation process completed!"
  ```
  
  **Nova server events (`openstack server event list --long`):**
  - No evacuation event recorded between 19:24:20 and 19:37:51 for either BFV VM
  - First Nova evacuation event: 19:37:51 (caad8b3d) and 19:37:59 (4efcc1bf)
  - compute_rebuild_instance completed: 19:38:21 and 19:38:55 (~1min after Nova accepted)
  
  **Volume state:**
  
  ```
  $ openstack volume show 35ab2ef8-5508-4401-bff4-06c2f119c225 | grep multiattach
  | multiattach | False |
  ```
  
  ## Note on log availability
  
  The Masakari, Nova conductor, Nova API, and Cinder API pod logs covering
  the critical 19:24–19:37 window were no longer available at the time of
  the investigation. The failed node was rebooted as part of the test
  recovery procedure, resulting in the loss of all pod logs on that node.
  The surviving nodes' pod logs had also rotated by the time the
  investigation was conducted (~2 days after the test).
  
  As a result, it was not possible to confirm from the logs whether:
  
  - Masakari submitted the initial evacuation request to Nova and received an error response (e.g., volume `in-use`, Nova rejecting the evacuate call), or
  - Masakari's internal evacuation loop detected the volume issue and deferred the request before reaching Nova
  
  The ~13-minute gap between Masakari's "Evacuation of instance started"
  log (19:24:20) and the first Nova server event (19:37:51) is consistent
  with Masakari's `FixedIntervalWithTimeoutLoopingCall` polling at
  `verify_interval=60s` - suggesting approximately 13 polling cycles
  elapsed before Masakari re-submitted the evacuation. The exact failure
  mode during the first attempt could not be confirmed due to log
  rotation.
  
  To reproduce and capture the full log sequence, the test should be
  repeated with extended log retention and pod log `--previous` capture
  immediately after the event.
  
  ## Root cause analysis
  
  Masakari's `EvacuateInstancesTask` calls Nova's evacuate API for all
  instances without first checking whether instances are boot-from-volume
  and whether their Cinder volumes need to be force-detached from the dead
  host.
  
  With `multiattach=False`, Nova cannot attach the volume to the
  destination host until it has detached it from the source. Since the
  source host is powered off, Nova cannot cleanly detach the iSCSI session
  - it requires either:
  
  1. The source hypervisor to come back online and clear the session, or
  2. A `os-force_detach` call to Cinder to forcibly clear the volume attachment
  
  Masakari currently does neither. It calls Nova's evacuate API and then
  polls for confirmation (`FixedIntervalWithTimeoutLoopingCall` with
  `verify_interval=60s`). When Nova cannot complete the evacuation (volume
  still `in-use`), Masakari eventually re-queues the evacuation after its
  internal retry logic (~13 minutes in our test), at which point it
  succeeds because enough time has passed for the volume state to change.
  
  ## Expected behavior
  
  Masakari should detect BFV instances and call `os-force_detach` on their
  Cinder volumes before or during the evacuation call to the Nova API.
  Since the backend is shared storage (Pure Storage FA, Ceph, etc.), the
  storage array can terminate the dead host's iSCSI/RBD session without
  requiring the hypervisor to be alive. This would allow Nova to complete
  the evacuation immediately, reducing BFV evacuation time from ~37
  minutes to under 2 minutes (matching ephemeral instances).
  
  ## Workaround
  
  Manually reset Cinder volume state immediately after host power-off:
  
  ```bash
  openstack volume set --state available <volume-id>
  ```
  
  Or enable `multiattach` on the Pure Storage volume types, which allows
  Nova to attach to the destination before detaching from the source:
  
  ```bash
  openstack volume type set --property multiattach='<is> True' <volume-type>
  ```
  
  ## Impact
  
  Any Masakari deployment that uses BFV instances with `multiattach=False`
  on a shared storage backend (Pure Storage, NetApp, etc.) will experience
  significantly delayed evacuation in the event of a hard host failure.
  This makes Masakari HA effectively unusable for BFV workloads without
  the multiattach workaround.
  
  ## References
  
  - https://canonical-openstack.readthedocs-hosted.com/en/latest/how-
  to/features/instance-recovery/#supplementary-information

** Description changed:

  [ Impact ]
  
  When Masakari handles a compute-host failure, a transient failure while
  querying Nova or Keystone can escape from an evacuation worker before the corresponding
  VM move is updated.
  
  The VM move remains in PENDING state, but Masakari can mark the overall
  host-failure notification as finished. Operators are therefore told that
  recovery completed even though an instance was not evacuated and may remain
  unavailable.
  
  The update ensures that these failures mark the VM move and host-recovery
  notification as failed instead of silently reporting successful recovery.
  
  [ Test Plan ]
  
  This test requires an OpenStack deployment with Masakari and at least two Nova
  compute hosts.
  
- 1. Install the release version of Masakari from the Resolute archive.
+ 1. Install the release version of Masakari from the archive.
  
  2. Create an ACTIVE instance on the first compute host and set its
     HA_Enabled property to True.
  
  3. While processing a compute-host failure notification, make the Nova
     server-detail request for that instance return HTTP 503.
  
  4. Confirm the existing bug:
  
     * the Masakari notification reaches FINISHED;
     * the instance has not been evacuated; and
     * the corresponding vmove remains PENDING without completion timestamps.
  
  5. Restore the compute host and Masakari host state.
  
  6. Install the Masakari packages from proposed and confirm the
     installed binary package version.
  
  7. Repeat steps 2 through 4 with the same injected Nova failure.
  
  8. Confirm the corrected behaviour:
  
     * the Masakari notification reaches FAILED;
     * the associated vmove records reach FAILED;
     * failed vmoves have an end time and error message; and
     * no vmove associated with the notification remains PENDING.
  
  9. Remove the injected Nova failure and perform another host evacuation with
     an ACTIVE HA-enabled instance.
  
  10. Confirm that the normal evacuation succeeds:
  
      * the notification reaches FINISHED;
      * the vmove reaches SUCCEEDED;
      * Nova reports the instance ACTIVE on the second compute;
      * the instance is running under libvirt on the second compute;
      * its Neutron port is ACTIVE and bound to the second compute; and
      * both compute services can be restored to the up/enabled state.
  
  [ Where problems could occur ]
  
  The change affects the compute-host recovery path, including instance lookup,
  instance locking and unlocking, and the final determination of whether host
  recovery succeeded.
  
  An error in this logic could incorrectly fail a successful evacuation, leave
  an instance locked, or allow a failed evacuation to be reported as successful.
  
  The added pending-vmove check could also expose an existing workflow or
  persistence race as a failed notification. The test plan therefore covers both
  the error path and a normal successful evacuation between two compute hosts.
  
  [ Other Info ]
  
  Upstream fix:
  
  https://opendev.org/openstack/masakari/commit/4e6dd2e645d71439eb289a1523a5d04c402e90bb
  
  The upstream source containing this fix uses a newer
  oslo.versionedobjects decorator API than the version available in Resolute/Noble.
- The backport retains Resolute's existing decorator API and includes only the
+ The backport retains Resolute/Noble's existing decorator API and includes only the
  behavioural changes required to fix this bug.
  
  --- previous description
  ## Summary
  
  Masakari fails to evacuate boot-from-volume (BFV) instances promptly
  during host failure when the Cinder volume type has `multiattach=False`.
  The evacuation is delayed by ~13 minutes before being re-queued, and
  completes only after the source hypervisor recovers and clears the iSCSI
  session, or after manual Cinder volume state reset.
  
  ## Environment
  
  - **Deployment method:** OpenStack Sunbeam
  - **OpenStack release:** 2024.1 (Nova 29.2.0)
  - **Masakari:** 2024.1/stable (masakari-k8s charm rev 137)
  - **Cinder backend:** Pure Storage FA iSCSI (PureISCSIDriver)
  - **Volume multiattach:** False (default)
  - **Cluster:** 3-node, all nodes are control + compute + storage
  
  ## Steps to reproduce
  
  1. Create a BFV instance using a Cinder volume with `multiattach=False` on a Pure Storage iSCSI backend
  2. Configure Masakari with a failover segment covering the host
  3. Hard power-off the host (IPMI/BMC hard power cut)
  4. Observe Masakari's evacuation behavior
  
  ## Observed behavior
  
  Masakari's `EvacuateInstancesTask` logs `"Evacuation of instance
  started"` for all instances, including BFV ones, at T+0. However:
  
  - **Ephemeral instances (4/6):** evacuated successfully within 2min 26s
  - **BFV instances (2/6):** no Nova server event recorded for ~13 minutes after Masakari triggered evacuation. Nova migration records are only created at T+13min. Actual rebuild completed in under 1 minute once Nova accepted the request.
  
  Total time from host failure to BFV instance availability on
  destination: **~37 minutes** (13min delay + wait for source node
  recovery to clear iSCSI session when powering on the host).
  
  ## Evidence
  
  **Masakari notification detail (EvacuateInstancesTask):**
  
  ```
  19:24:20 - "Evacuation of instance started: 'caad8b3d-c2c3-43ec-845f-7c5f056ebcd4'"
  19:24:20 - "Evacuation of instance started: '4efcc1bf-8c3b-4467-8f75-420ff0b3863a'"
  19:26:47 - "Evacuation process completed!"
  ```
  
  **Nova server events (`openstack server event list --long`):**
  - No evacuation event recorded between 19:24:20 and 19:37:51 for either BFV VM
  - First Nova evacuation event: 19:37:51 (caad8b3d) and 19:37:59 (4efcc1bf)
  - compute_rebuild_instance completed: 19:38:21 and 19:38:55 (~1min after Nova accepted)
  
  **Volume state:**
  
  ```
  $ openstack volume show 35ab2ef8-5508-4401-bff4-06c2f119c225 | grep multiattach
  | multiattach | False |
  ```
  
  ## Note on log availability
  
  The Masakari, Nova conductor, Nova API, and Cinder API pod logs covering
  the critical 19:24–19:37 window were no longer available at the time of
  the investigation. The failed node was rebooted as part of the test
  recovery procedure, resulting in the loss of all pod logs on that node.
  The surviving nodes' pod logs had also rotated by the time the
  investigation was conducted (~2 days after the test).
  
  As a result, it was not possible to confirm from the logs whether:
  
  - Masakari submitted the initial evacuation request to Nova and received an error response (e.g., volume `in-use`, Nova rejecting the evacuate call), or
  - Masakari's internal evacuation loop detected the volume issue and deferred the request before reaching Nova
  
  The ~13-minute gap between Masakari's "Evacuation of instance started"
  log (19:24:20) and the first Nova server event (19:37:51) is consistent
  with Masakari's `FixedIntervalWithTimeoutLoopingCall` polling at
  `verify_interval=60s` - suggesting approximately 13 polling cycles
  elapsed before Masakari re-submitted the evacuation. The exact failure
  mode during the first attempt could not be confirmed due to log
  rotation.
  
  To reproduce and capture the full log sequence, the test should be
  repeated with extended log retention and pod log `--previous` capture
  immediately after the event.
  
  ## Root cause analysis
  
  Masakari's `EvacuateInstancesTask` calls Nova's evacuate API for all
  instances without first checking whether instances are boot-from-volume
  and whether their Cinder volumes need to be force-detached from the dead
  host.
  
  With `multiattach=False`, Nova cannot attach the volume to the
  destination host until it has detached it from the source. Since the
  source host is powered off, Nova cannot cleanly detach the iSCSI session
  - it requires either:
  
  1. The source hypervisor to come back online and clear the session, or
  2. A `os-force_detach` call to Cinder to forcibly clear the volume attachment
  
  Masakari currently does neither. It calls Nova's evacuate API and then
  polls for confirmation (`FixedIntervalWithTimeoutLoopingCall` with
  `verify_interval=60s`). When Nova cannot complete the evacuation (volume
  still `in-use`), Masakari eventually re-queues the evacuation after its
  internal retry logic (~13 minutes in our test), at which point it
  succeeds because enough time has passed for the volume state to change.
  
  ## Expected behavior
  
  Masakari should detect BFV instances and call `os-force_detach` on their
  Cinder volumes before or during the evacuation call to the Nova API.
  Since the backend is shared storage (Pure Storage FA, Ceph, etc.), the
  storage array can terminate the dead host's iSCSI/RBD session without
  requiring the hypervisor to be alive. This would allow Nova to complete
  the evacuation immediately, reducing BFV evacuation time from ~37
  minutes to under 2 minutes (matching ephemeral instances).
  
  ## Workaround
  
  Manually reset Cinder volume state immediately after host power-off:
  
  ```bash
  openstack volume set --state available <volume-id>
  ```
  
  Or enable `multiattach` on the Pure Storage volume types, which allows
  Nova to attach to the destination before detaching from the source:
  
  ```bash
  openstack volume type set --property multiattach='<is> True' <volume-type>
  ```
  
  ## Impact
  
  Any Masakari deployment that uses BFV instances with `multiattach=False`
  on a shared storage backend (Pure Storage, NetApp, etc.) will experience
  significantly delayed evacuation in the event of a hard host failure.
  This makes Masakari HA effectively unusable for BFV workloads without
  the multiattach workaround.
  
  ## References
  
  - https://canonical-openstack.readthedocs-hosted.com/en/latest/how-
  to/features/instance-recovery/#supplementary-information

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2158101

Title:
  [SRU] Masakari marks host recovery finished with pending VM moves
  after Nova or Keystone API failure

Status in Ubuntu Cloud Archive caracal series:
  New
Status in Ubuntu Cloud Archive epoxy series:
  New
Status in Ubuntu Cloud Archive gazpacho series:
  New
Status in Ubuntu Cloud Archive hibiscus series:
  New
Status in masakari:
  Fix Committed
Status in OpenStack Snap:
  Incomplete
Status in masakari package in Ubuntu:
  Fix Released
Status in masakari source package in Noble:
  New
Status in masakari source package in Resolute:
  In Progress
Status in masakari source package in Stonking:
  Fix Released

Bug description:
  [ Impact ]

  When Masakari handles a compute-host failure, a transient failure while
  querying Nova or Keystone can escape from an evacuation worker before the corresponding
  VM move is updated.

  The VM move remains in PENDING state, but Masakari can mark the overall
  host-failure notification as finished. Operators are therefore told that
  recovery completed even though an instance was not evacuated and may remain
  unavailable.

  The update ensures that these failures mark the VM move and host-recovery
  notification as failed instead of silently reporting successful recovery.

  [ Test Plan ]

  This test requires an OpenStack deployment with Masakari and at least two Nova
  compute hosts.

  1. Install the release version of Masakari from the archive.

  2. Create an ACTIVE instance on the first compute host and set its
     HA_Enabled property to True.

  3. While processing a compute-host failure notification, make the Nova
     server-detail request for that instance return HTTP 503.

  4. Confirm the existing bug:

     * the Masakari notification reaches FINISHED;
     * the instance has not been evacuated; and
     * the corresponding vmove remains PENDING without completion timestamps.

  5. Restore the compute host and Masakari host state.

  6. Install the Masakari packages from proposed and confirm the
     installed binary package version.

  7. Repeat steps 2 through 4 with the same injected Nova failure.

  8. Confirm the corrected behaviour:

     * the Masakari notification reaches FAILED;
     * the associated vmove records reach FAILED;
     * failed vmoves have an end time and error message; and
     * no vmove associated with the notification remains PENDING.

  9. Remove the injected Nova failure and perform another host evacuation with
     an ACTIVE HA-enabled instance.

  10. Confirm that the normal evacuation succeeds:

      * the notification reaches FINISHED;
      * the vmove reaches SUCCEEDED;
      * Nova reports the instance ACTIVE on the second compute;
      * the instance is running under libvirt on the second compute;
      * its Neutron port is ACTIVE and bound to the second compute; and
      * both compute services can be restored to the up/enabled state.

  [ Where problems could occur ]

  The change affects the compute-host recovery path, including instance lookup,
  instance locking and unlocking, and the final determination of whether host
  recovery succeeded.

  An error in this logic could incorrectly fail a successful evacuation, leave
  an instance locked, or allow a failed evacuation to be reported as successful.

  The added pending-vmove check could also expose an existing workflow or
  persistence race as a failed notification. The test plan therefore covers both
  the error path and a normal successful evacuation between two compute hosts.

  [ Other Info ]

  Upstream fix:

  https://opendev.org/openstack/masakari/commit/4e6dd2e645d71439eb289a1523a5d04c402e90bb

  The upstream source containing this fix uses a newer
  oslo.versionedobjects decorator API than the version available in Resolute/Noble.
  The backport retains Resolute/Noble's existing decorator API and includes only the
  behavioural changes required to fix this bug.

  --- previous description
  ## Summary

  Masakari fails to evacuate boot-from-volume (BFV) instances promptly
  during host failure when the Cinder volume type has
  `multiattach=False`. The evacuation is delayed by ~13 minutes before
  being re-queued, and completes only after the source hypervisor
  recovers and clears the iSCSI session, or after manual Cinder volume
  state reset.

  ## Environment

  - **Deployment method:** OpenStack Sunbeam
  - **OpenStack release:** 2024.1 (Nova 29.2.0)
  - **Masakari:** 2024.1/stable (masakari-k8s charm rev 137)
  - **Cinder backend:** Pure Storage FA iSCSI (PureISCSIDriver)
  - **Volume multiattach:** False (default)
  - **Cluster:** 3-node, all nodes are control + compute + storage

  ## Steps to reproduce

  1. Create a BFV instance using a Cinder volume with `multiattach=False` on a Pure Storage iSCSI backend
  2. Configure Masakari with a failover segment covering the host
  3. Hard power-off the host (IPMI/BMC hard power cut)
  4. Observe Masakari's evacuation behavior

  ## Observed behavior

  Masakari's `EvacuateInstancesTask` logs `"Evacuation of instance
  started"` for all instances, including BFV ones, at T+0. However:

  - **Ephemeral instances (4/6):** evacuated successfully within 2min 26s
  - **BFV instances (2/6):** no Nova server event recorded for ~13 minutes after Masakari triggered evacuation. Nova migration records are only created at T+13min. Actual rebuild completed in under 1 minute once Nova accepted the request.

  Total time from host failure to BFV instance availability on
  destination: **~37 minutes** (13min delay + wait for source node
  recovery to clear iSCSI session when powering on the host).

  ## Evidence

  **Masakari notification detail (EvacuateInstancesTask):**

  ```
  19:24:20 - "Evacuation of instance started: 'caad8b3d-c2c3-43ec-845f-7c5f056ebcd4'"
  19:24:20 - "Evacuation of instance started: '4efcc1bf-8c3b-4467-8f75-420ff0b3863a'"
  19:26:47 - "Evacuation process completed!"
  ```

  **Nova server events (`openstack server event list --long`):**
  - No evacuation event recorded between 19:24:20 and 19:37:51 for either BFV VM
  - First Nova evacuation event: 19:37:51 (caad8b3d) and 19:37:59 (4efcc1bf)
  - compute_rebuild_instance completed: 19:38:21 and 19:38:55 (~1min after Nova accepted)

  **Volume state:**

  ```
  $ openstack volume show 35ab2ef8-5508-4401-bff4-06c2f119c225 | grep multiattach
  | multiattach | False |
  ```

  ## Note on log availability

  The Masakari, Nova conductor, Nova API, and Cinder API pod logs
  covering the critical 19:24–19:37 window were no longer available at
  the time of the investigation. The failed node was rebooted as part of
  the test recovery procedure, resulting in the loss of all pod logs on
  that node. The surviving nodes' pod logs had also rotated by the time
  the investigation was conducted (~2 days after the test).

  As a result, it was not possible to confirm from the logs whether:

  - Masakari submitted the initial evacuation request to Nova and received an error response (e.g., volume `in-use`, Nova rejecting the evacuate call), or
  - Masakari's internal evacuation loop detected the volume issue and deferred the request before reaching Nova

  The ~13-minute gap between Masakari's "Evacuation of instance started"
  log (19:24:20) and the first Nova server event (19:37:51) is
  consistent with Masakari's `FixedIntervalWithTimeoutLoopingCall`
  polling at `verify_interval=60s` - suggesting approximately 13 polling
  cycles elapsed before Masakari re-submitted the evacuation. The exact
  failure mode during the first attempt could not be confirmed due to
  log rotation.

  To reproduce and capture the full log sequence, the test should be
  repeated with extended log retention and pod log `--previous` capture
  immediately after the event.

  ## Root cause analysis

  Masakari's `EvacuateInstancesTask` calls Nova's evacuate API for all
  instances without first checking whether instances are boot-from-
  volume and whether their Cinder volumes need to be force-detached from
  the dead host.

  With `multiattach=False`, Nova cannot attach the volume to the
  destination host until it has detached it from the source. Since the
  source host is powered off, Nova cannot cleanly detach the iSCSI
  session - it requires either:

  1. The source hypervisor to come back online and clear the session, or
  2. A `os-force_detach` call to Cinder to forcibly clear the volume attachment

  Masakari currently does neither. It calls Nova's evacuate API and then
  polls for confirmation (`FixedIntervalWithTimeoutLoopingCall` with
  `verify_interval=60s`). When Nova cannot complete the evacuation
  (volume still `in-use`), Masakari eventually re-queues the evacuation
  after its internal retry logic (~13 minutes in our test), at which
  point it succeeds because enough time has passed for the volume state
  to change.

  ## Expected behavior

  Masakari should detect BFV instances and call `os-force_detach` on
  their Cinder volumes before or during the evacuation call to the Nova
  API. Since the backend is shared storage (Pure Storage FA, Ceph,
  etc.), the storage array can terminate the dead host's iSCSI/RBD
  session without requiring the hypervisor to be alive. This would allow
  Nova to complete the evacuation immediately, reducing BFV evacuation
  time from ~37 minutes to under 2 minutes (matching ephemeral
  instances).

  ## Workaround

  Manually reset Cinder volume state immediately after host power-off:

  ```bash
  openstack volume set --state available <volume-id>
  ```

  Or enable `multiattach` on the Pure Storage volume types, which allows
  Nova to attach to the destination before detaching from the source:

  ```bash
  openstack volume type set --property multiattach='<is> True' <volume-type>
  ```

  ## Impact

  Any Masakari deployment that uses BFV instances with
  `multiattach=False` on a shared storage backend (Pure Storage, NetApp,
  etc.) will experience significantly delayed evacuation in the event of
  a hard host failure. This makes Masakari HA effectively unusable for
  BFV workloads without the multiattach workaround.

  ## References

  - https://canonical-openstack.readthedocs-hosted.com/en/latest/how-
  to/features/instance-recovery/#supplementary-information

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/caracal/+bug/2158101/+subscriptions