[Bug 2146964] Re: [SRU] Prevent masakari HA from creating duplicated notifications

Alan Baghumian 2146964 at bugs.launchpad.net
Sat Sep 19 15:16:30 UTC 2026 

Hello Nick Thank you so much for helping me with the SRU uploads.

Resolute Test Results:

0. Deployed OpenStack Resolute/Gazpacho (2026.1) using OpenStack Ansible
(1) with HA Masakari. This officially supports Resolute.

root at os-control-0-masakari-api-container-ac035ee1:~# dpkg -l | grep masakari
ii  masakari-common                   21.0.0-0ubuntu1                    all
ii  python3-masakari                  21.0.0-0ubuntu1                    all

root at os-control-1-masakari-api-container-38c15660:~# dpkg -l | grep masakari
ii  masakari-common                   21.0.0-0ubuntu1                    all
ii  python3-masakari                  21.0.0-0ubuntu1                    all

root at os-control-2-masakari-api-container-57faa954:~# dpkg -l | grep masakari
ii  masakari-common                   21.0.0-0ubuntu1                    all
ii  python3-masakari                  21.0.0-0ubuntu1                    all


1. Get it configured

openstack segment create nova auto COMPUTE
for x in {0..2}; do openstack segment host create os-compute-${x} COMPUTE SSH nova; done

2. Enabled resolute-proposed and Upgraded masakari packages on all 3
containers and rebooted them:

root at os-control-0-masakari-api-container-ac035ee1:~# dpkg -l | grep masakari
ii  masakari-common                   21.0.0-0ubuntu1.1                  all
ii  python3-masakari                  21.0.0-0ubuntu1.1                  all

3. Utilized the reproducer script (2) and executed it a few times (logs
attached).

3. Check the notification list output and there are no duplicates:

root at os-control-0-utility-container-1f4084f7:~# openstack notification list --sort-column generated_time -f value| tail -n 30
47b32625-d7cc-4c61-9def-ee261df6563f 2026-09-19T14:49:20.000000 finished COMPUTE_HOST 331a1679-8333-4ff3-a8ca-cfbf9c4ead5b {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'}
fe9ee659-c4a4-464e-a84c-24a184e33d23 2026-09-19T15:03:12.000000 finished COMPUTE_HOST 1d1e9a94-9762-4526-812f-d18538a83443 {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'}


(1) https://docs.openstack.org/openstack-ansible/2026.1/
(2) https://launchpadlibrarian.net/849768758/lp2028450-reproducer.bash


** Attachment added: "Resolute Verification Results"
   https://bugs.launchpad.net/cloud-archive/ussuri/+bug/2146964/+attachment/6001200/+files/resolute-verification-results.txt

** Tags removed: verification-needed-resolute
** Tags added: verification-done-resolute

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2146964

Title:
  [SRU] Prevent masakari HA from creating duplicated notifications

Status in Ubuntu Cloud Archive:
  In Progress
Status in Ubuntu Cloud Archive caracal series:
  Fix Committed
Status in Ubuntu Cloud Archive dalmatian series:
  Won't Fix
Status in Ubuntu Cloud Archive epoxy series:
  Fix Committed
Status in Ubuntu Cloud Archive flamingo series:
  Fix Committed
Status in Ubuntu Cloud Archive gazpacho series:
  Fix Committed
Status in Ubuntu Cloud Archive ussuri series:
  Won't Fix
Status in Ubuntu Cloud Archive yoga series:
  Fix Committed
Status in masakari:
  Fix Committed
Status in masakari package in Ubuntu:
  Fix Released
Status in masakari source package in Focal:
  Won't Fix
Status in masakari source package in Jammy:
  Fix Committed
Status in masakari source package in Noble:
  Fix Committed
Status in masakari source package in Questing:
  Won't Fix
Status in masakari source package in Resolute:
  Fix Committed
Status in masakari source package in Stonking:
  Fix Released

Bug description:
  [ Impact ]

   * Older versions of Masakari are lacking a proper mechanism to prevent
     concurrent record inserts into the database, resulting duplicated
     event processing that can potentially lead to disastrous outcomes.
     These include simultanious server migration attempts due to duplicated
     host notifications that can cause failures and database inconsistencies
     which are not always easy to fix.

   * Affected versios include:

     - Focal/Ussuri 9.0.0-0ubuntu0.20.04.5 / 9.0.0-0ubuntu0.20.04.5~cloud0 (UCA) (Won't Fix - Code too old)
     - Jammy/Yoga 13.0.0-0ubuntu1 / 13.0.0-0ubuntu1~cloud0 (UCA)
     - Jammy/Caracal 17.0.0-0ubuntu1 /	17.0.0-0ubuntu1~cloud0 (UCA)
     - Noble 17.0.0-0ubuntu1
     - Resolute 21.0.0-0ubuntu1
     - Stonking 21.0.0-0ubuntu1

   * This is a known issue and reported by many users also presented in
     LP#2028450.

   * This requires an HA Masakari deployment without a coordinator
     configured.

   * None of the current Charmed Masakari deployments support using a
     coordinator and are all affected.

  [ Test Plan ]

   * Testing requires an OpenStack environment deployed with HA Masakari.
     Masakari deployed in HA mode is a hard requirement as the issue only
     affects HA deployments due to workers competing with each other.

   * The LP#2028450 bug includes a reproducer script (2), which is handy
     to reproduce the issue as well as validate the fix.

   * The reproducer script simulates concurrent insertion situations
     using OpenStack CLI.
     
   * After running the reproducer script, execute the following command:
   
  openstack notification list --sort-column generated_time --insecure | tail -n 30
   
   * Bad Case: Look for COMPUTE_HOST entries that have close timestamps and pay attention
     to the state column. The following is an example of a problematic state with
     two entires created with close proximity affecting the same host and both
     and not ignored:
     
  | 9a06d57e-4fb3-4d44-bea3-c3d6f5ee3ea8 | 2026-04-22T17:41:47.000000 | finished | COMPUTE_HOST | 1cb67d44-4a3f-4855-bf3e-960aec3aa079
  | 60075e3c-1d85-4825-86d6-071d345a98fb | 2026-04-22T17:41:47.000000 | finished | COMPUTE_HOST | 1cb67d44-4a3f-4855-bf3e-960aec3aa079

   * Good Case: There should not be duplicated entries with a close proximity:
   
  | 06a5422f-0ea6-46a1-b6b7-a171158f7b57 | 2026-04-22T18:00:47.000000 | finished | COMPUTE_HOST | 1cb67d44-4a3f-4855-bf3e-960aec3aa079

  
  [ Where problems could occur ]

   * The nature if this change is very simple. In the HA API, the
     create notification function will be modified to introduce an
     artificial pause, significantly reducing the chance of
     concurrent insertion if event records in database.

     This will only be invoked if coordination back-end is not
     configured. Please see (3) for more details:

     if not CONF.coordination.backend_url:
             time.sleep(random.uniform(1, 5))

   * The probable risk here is a delay between 1-5 seconds during
     event creation and further processing. These events happen
     for example when a compute node becomes inaccessible and the
     instances need to be evacuated and launched on a new node.
     In other words the recovery will be delayed for 1-5 seconds.

  [ Other Info ]

   * The proposed changes have been merged upstream (4).

  (1) https://bugs.launchpad.net/masakari/+bug/2028450
  (2) https://launchpadlibrarian.net/849768758/lp2028450-reproducer.bash
  (3) https://review.opendev.org/c/openstack/masakari/+/978343/3/masakari/ha/api.py
  (4) https://review.opendev.org/c/openstack/masakari/+/978343

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2146964/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list