[Bug 2146964] Re: [SRU] Prevent masakari HA from creating duplicated notifications

Matthew Ruffell 2146964 at bugs.launchpad.net
Fri Jul 3 05:17:32 UTC 2026 

Okay, Seyeong and I had a look during our Sponsorship Office Hours:

In your [Impact] section, you mention "disastrous outcomes" but you don't tell
us what they outcomes are, or how they happen.

The [Testcase] section needs some additional details. Imagine you are writing
a test case to be followed by someone who has some understanding of OpenStack.
Could they follow your testcase to verify a package in -proposed themselves?
What extra details do they need to know? What does a "bad" case look like, and
what does a "good" case look like?

For your PPAs, you just need one, internally you can have multiple packages of
the same name, just that they need to target different series. So as long as
you keep the questing package for questing, and the noble package for noble,
they can all live in the same ppa.

Usually, as part of maintaining OpenStack packages, we aim to land the patch in
the upstream backport branches, so please target backports to all supported
OpenStack releases:
> If any affected OpenStack release is still supported upstream, it is required
> that the patch be merged in the stable branch for that release before an SRU 
> can proceed.

For the patches:

In Debian/changelog:

Usually we would have a format like this:

+masakari (17.0.0-0ubuntu1.24.04.1) noble; urgency=medium
+
+  * If coordination is NOT configured, synchronized becomes effectively
+    a no-op across multiple API workers, which can lead to duplicate
+    notifications due to race conditions. Introduce a small randomized
+    pause to reduce probability of concurrent inserts. (LP: #2028450)
+    - d/p/lp2146964-masakari-ha-no-duplicated-notifications.patch
+
+ -- Alan Baghumian <alan.baghumian at canonical.com>  Mon, 06 Apr 2026 11:31:00 -0700

Or this is okay too:

+masakari (17.0.0-0ubuntu1.24.04.1) noble; urgency=medium
+
+  * d/p/lp2146964-masakari-ha-no-duplicated-notifications.patch:
+    If coordination is NOT configured, synchronized becomes effectively
+    a no-op across multiple API workers, which can lead to duplicate
+    notifications due to race conditions. Introduce a small randomized
+    pause to reduce probability of concurrent inserts. (LP: #2028450)
+
+ -- Alan Baghumian <alan.baghumian at canonical.com>  Mon, 06 Apr 2026 11:31:00 -0700

but I usually prefer the former.

Your versions are incorrect, please check:

https://ubuntu.com/project/docs/how-ubuntu-is-made/concepts/version-
strings/#version-adding-a-change-in-ubuntu-as-a-stable-release-update

But for reference, they would be:

stonking: 21.0.0-0ubuntu2
resolute: 21.0.0-0ubuntu1.1
questing: 20.0.0-0ubuntu1.1 (but lets be real, questing will EOL before this is
ready so target flamingo UCA with 20.0.0-0ubuntu1~cloud1)
epoxy: 19.0.0-0ubuntu1~cloud1
noble: 17.0.0-0ubuntu1.1
jammy: 13.0.0-0ubuntu1.1

As Vladimir mentioned, please add some Dep3 tags, mostly, the:

Origin: upstream/backport, https://opendev.org/openstack/masakari/commit/1850c050ef716f80706dc29988bee901950f5d2c
Bug: https://bugs.launchpad.net/masakari/+bug/2028450
Bug-Ubuntu: https://bugs.launchpad.net/masakari/+bug/2146964 

Make sure to swap between "upstream" for cherry picks and "backport" if the
patch needed any changes.

If you could follow:

https://documentation.ubuntu.com/se-docs/en/latest/se-
sponsorship/ubuntu-cloud-archive-sru/

and open merge requests against the correct git branches for:

https://git.launchpad.net/~ubuntu-openstack-dev/ubuntu/+source/masakari

mapped like this:

stonking -> master
resolute -> stable/2026.1
questing / flamingo -> stable/2025.2
epoxy -> stable/2025.1
noble: stable/2024.1
jammy: stable/yoga

it would help a lot.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2146964

Title:
  [SRU] Prevent masakari HA from creating duplicated notifications

Status in Ubuntu Cloud Archive:
  In Progress
Status in Ubuntu Cloud Archive caracal series:
  In Progress
Status in Ubuntu Cloud Archive dalmatian series:
  New
Status in Ubuntu Cloud Archive epoxy series:
  New
Status in Ubuntu Cloud Archive flamingo series:
  In Progress
Status in Ubuntu Cloud Archive gazpacho series:
  In Progress
Status in Ubuntu Cloud Archive ussuri series:
  Won't Fix
Status in Ubuntu Cloud Archive yoga series:
  In Progress
Status in masakari:
  Fix Committed
Status in masakari package in Ubuntu:
  In Progress
Status in masakari source package in Focal:
  Won't Fix
Status in masakari source package in Jammy:
  In Progress
Status in masakari source package in Noble:
  In Progress
Status in masakari source package in Questing:
  In Progress
Status in masakari source package in Resolute:
  In Progress
Status in masakari source package in Stonking:
  In Progress

Bug description:
  [ Impact ]

   * Older versions of Masakari are lacking a proper mechanism to prevent
     concurrent record inserts into the database, resulting duplicated
     event processing that can potentially lead to disastrous outcomes.

   * Affected versios include:

     - Focal/Ussuri 9.0.0-0ubuntu0.20.04.5 / 9.0.0-0ubuntu0.20.04.5~cloud0 (UCA)
     - Jammy/Yoga 13.0.0-0ubuntu1 / 13.0.0-0ubuntu1~cloud0 (UCA)
     - Jammy/Caracal 17.0.0-0ubuntu1 /	17.0.0-0ubuntu1~cloud0 (UCA)
     - Noble 17.0.0-0ubuntu1
     - Resolute 21.0.0-0ubuntu1
     - Stonking 21.0.0-0ubuntu1

   * This is a known issue and reported by many users also presented in
     LP#2028450.

   * This requires an HA Masakari deployment without a coordinator
     configured.

   * None of the current Charmed Masakari deployments support using a
     coordinator and are all affected.

  [ Test Plan ]

   * Testing requires an OpenStack environment deployed with HA Masakari.
     Masakari deployed in HA mode is a hard requirement as the issue only
     affects HA deployments due to workers competing with each other.

   * Juju bundles will be crafted and attached to this SRU bug to make
     reproducer deployments easier. The plan is to cover all affected
     versions. Detailed instructions will be provided.

   * The LP#2028450 bug includes a reproducer script (2), which is handy
     to reproduce the issue as well as validate the fix.

   * The reproducer script simulates concurrent insertion situations
     using OpenStack CLI.

  [ Where problems could occur ]

   * The nature if this change is very simple. In the HA API, the
     create notification function will be modified to introduce an
     artificial pause, significantly reducing the chance of
     concurrent insertion if event records in database.

     This will only be invoked if coordination back-end is not
     configured. Please see (3) for more details:

     if not CONF.coordination.backend_url:
             time.sleep(random.uniform(1, 5))

   * The probable risk here is a delay between 1-5 seconds during
     event creation and further processing. These events happen
     for example when a compute node becomes inaccessible and the
     instances need to be evacuated and launched on a new node.
     In other words the recovery will be delayed for 1-5 seconds.

  [ Other Info ]

   * There is currently an effort to merge this change upstream (4).

  (1) https://bugs.launchpad.net/masakari/+bug/2028450
  (2) https://launchpadlibrarian.net/849768758/lp2028450-reproducer.bash
  (3) https://review.opendev.org/c/openstack/masakari/+/978343/3/masakari/ha/api.py
  (4) https://review.opendev.org/c/openstack/masakari/+/978343

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2146964/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list