[Bug 1988457] Re: [SRU] ovsdbapp can time out on raft leadership change

[Ubuntu Foundations Team Bug Bot]

The attachment "lp1988457-jammy.debdiff" seems to be a debdiff.  The
ubuntu-sponsors team has been subscribed to the bug report so that they
can review and hopefully sponsor the debdiff.  If the attachment isn't a
patch, please remove the "patch" flag from the attachment, remove the
"patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe
the team.

[This is an automated message performed by a Launchpad user owned by
~brian-murray, for any issue please contact him.]

** Tags added: patch

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1988457

Title:
  [SRU] ovsdbapp can time out on raft leadership change

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive yoga series:
  New
Status in ovsdbapp:
  Fix Released
Status in python-ovsdbapp package in Ubuntu:
  New
Status in python-ovsdbapp source package in Jammy:
  New

Bug description:
  When raft leadership changes, any leader-only connections will be
  disconnected and will need to reconnect to the new leader. When this
  happens, the IDL will return a txn status of TRY_AGAIN. The current
  code tries to do an exponential backoff with sleep() due to an issue
  where those can be spammed 1000s of times a second. This sleep also
  prevents reconnecting quickly because idl.run() is not called rapidly
  and can lead to timeouts.

  --------------------------------------------------------------------------------
  SRU TEMPLATE:

  [Impact]

  Please see original bug description. What i can add to this is that
  what we saw in production as a consequence of this was that ovsdbapp
  transactions would fail after a timeout and ovsdbapp would then end up
  in a retry sequence such that the transations would not get retried
  and vm tap devices would not get deleted from ovs when a vm was
  deleted. The result was a build up of "stale" tap devices on br-int
  (visible as "No such device" entries in ovs-vsctl show).

  [Test Plan]

  * Deploy OpenStack Jammy (Yoga) with ml2-ovn
  * Spawn several vms
  * Trigger many ovn-central db leadership switches by restarting ovn-central units in rotation leaving enough between each for a new leader to be elected.
  * Delete the vms and create a load more while leaders are being re-elected.
  * First check that /var/log/nova/nova-compute.log does not contain the "OVSDB transaction returned TRY_AGAIN" message over and over then also check that ovs-vsctl show does not contain any "stale" ports with messages like the following:

      Port tapa5d45fc6-02
          Interface tapa5d45fc6-02
              error: "could not open network device tapa5d45fc6-02 (No such device)"

  
  [Regression Potential]
  This patch is not expected to introduce any regressions.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1988457/+subscriptions