[Bug 2115328] Re: [uca][zed][antelope][bobcat] Upgrade failure due to missing patches

Thu Sep 17 09:53:13 UTC 2026

Verification performed in a fresh three-node Ubuntu 22.04.5 LTS
hyperconverged regress-stack lab, upgrading from Antelope to Bobcat. One
Cirros VM was pinned to each compute before the upgrade. All three passed
DNS, east-west, floating-IP, and routed traffic checks. Each host's
Southbound flow query found `flags.from_ctrl` and both ingress and egress
`ct_commit_nat` rules.

The Bobcat candidate OVN packages were
23.09.3-0ubuntu0.23.10.1~cloud1 from jammy-proposed/bobcat; OVS was
3.2.2-0ubuntu0.23.10.1~cloud0. Automatic package service actions were
suppressed. Each stopped controller was started immediately after package
installation. All three controllers ran the new version before OVN
databases/northd were restarted one node at a time. Guest floating-IP
checks passed between central-node restarts. OpenStack packages, database
schemas, and services were upgraded afterward.

After the upgrade, `regress-stack ready` passed. All three hosts retained
the target logical flows, active OVN services, and installed `br-int`
flows. The original three Cirros VMs retained their IDs and boot IDs and
passed DNS, east-west, and floating-IP checks after the upgrade and after
Tempest. The continuous flow watchers recorded no missing target rule.

The focused Tempest Nova server API and Neutron network API run selected
122 tests with one worker: 120 passed, 2 skipped, 0 failed (Nova: 57
passed, 2 skipped; Neutron: 63 passed). The earlier Bobcat Tempest failure
was not reproduced. The broad OVN parser-log scan retained one
`DHCPv6_Options` warning from the old Antelope northd process on each of
hosts 2 and 3 during the mixed-version transition. No controller parser
errors or subsequent warning from the new Bobcat northd processes were
found. The final Nova online-data-migration command exited 0 but reported
one matched `populate_instance_compute_id` row with none migrated; that
result is retained in the evidence.

STEP 1: ANTELOPE BASELINE AND CANDIDATE PROVENANCE
  All three hosts: ovn-common/central/host
    23.03.1-1ubuntu0.23.04.1~cloud4 (jammy-proposed/antelope)
    openvswitch-switch 3.1.3-0ubuntu0.23.04.1~cloud2
    nova-compute 3:27.4.0-0ubuntu1~cloud0
    neutron-server 2:22.1.0-0ubuntu1~cloud1
  regress-stack ready: passed
  Three Cirros VMs: ACTIVE, one pinned to each compute
  Guest DNS, cross-host ping, floating-IP ping, routed traceroute: passed
  Per-host target flow gate: true/true/true
  Per-host parser scan: empty/empty/empty
  Bobcat package simulation selected OVN
    23.09.3-0ubuntu0.23.10.1~cloud1 from jammy-proposed/bobcat
    and OVS 3.2.2-0ubuntu0.23.10.1~cloud0; no removals.

STEP 2: ROLLING UPGRADE AND MIGRATIONS
  Package service actions suppressed. Each stopped controller was started
  immediately after its package install; old northd and DB PIDs stayed
  unchanged until rolling central restarts.
  All three controllers were new before restarting central hosts 1, 2, 3.
  OVN DB quorum and 3/3 floating-IP pings passed at each checkpoint.
  One central helper reported failure on host 3 because its local follower
  rejected a leader-only CLI query. Manual checks confirmed changed PIDs,
  active services, cluster membership/quorum, and passing guest pings.
  OpenStack packages upgraded afterward; dpkg --audit empty on all hosts.
  Schema migrations and Neutron offline-migration check passed.
  Cinder online migrations: exit 0, none needed.
  Nova bounded online migration: exit 1 after migrating rows.
  Nova final unbounded online migration: exit 0; one
    populate_instance_compute_id row matched, zero migrated.

STEP 3: SERVICE, LOGICAL FLOW, AND GUEST GATES
  All three hosts: ovn-common/central/host
    23.09.3-0ubuntu0.23.10.1~cloud1
    openvswitch-switch 3.2.2-0ubuntu0.23.10.1~cloud0
    nova-compute 3:28.2.0-0ubuntu1~cloud0
    neutron-server 2:23.1.0-0ubuntu1~cloud1
  regress-stack ready before and after Tempest: passed.
  Target logical flows, per host (DNS, ingress NAT, egress NAT):
    hc1 1,1,1; hc2 1,1,1; hc3 1,1,1
  br-int OpenFlow dump lines: hc1 657; hc2 657; hc3 667
  Continuous flow watchers: no MISSING sample on any host.
  Original guest IDs, port IDs, boot IDs: unchanged; all three ACTIVE.
  Guest DNS, cross-host ping, floating-IP ping, routed traceroute: passed
    before and after Tempest.
  Nova compute disk_available_least after Tempest: 25 GiB on each host.
  OOM check after Tempest: empty.
  Broad historical parser scan: hc1 empty, hc2 one warning, hc3 one warning.
    At 09:03:36/38 UTC, old Antelope northd warned about a
    DHCPv6_Options domain type during the mixed-version transition.
    New Bobcat northd processes started 09:04:15/41 UTC. No later warning
    and no controller parser error found; the broad scan remains failed.

STEP 4: TEMPEST RESULT
  Command: tempest run --load-list nova-neutron-tests.txt --concurrency 1
  Selection regex: tempest.api.compute.servers.test_servers|
    tempest.api.network.test_networks
  Selected test-list lines: 122; exit 0.
  Nova server API: 57 passed, 2 skipped, 0 failed.
  Neutron network API: 63 passed, 0 failed.
  Total: 120 passed, 2 skipped, 0 failed in 1627 seconds.
  Tempest failure list empty. No Tempest server or network left.
  SHA256 tempest-last.txt:
    b8b5f4f53a852a5a0696b78065a88163cd3c48198a227138af851b182c976e05

** Tags removed: verification-bobcat-needed
** Tags added: verification-bobcat-done

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2115328

Title:
  [uca][zed][antelope][bobcat] Upgrade failure due to missing patches

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive antelope series:
  Fix Committed
Status in Ubuntu Cloud Archive bobcat series:
  Fix Committed
Status in Ubuntu Cloud Archive zed series:
  Fix Committed
Status in ovn package in Ubuntu:
  Invalid

Bug description:
  [Impact]

  Updated OVN 22.03 packages in Jammy generate Southbound logical flows using
  the ct_commit_nat action and flags.from_ctrl field. The original OVN
  22.09.1 packages in Zed do not understand these constructs.

  During the upgrade, ovn-controller rejects the affected logical flows,
  logging:

    error parsing actions "ct_commit_nat;": Syntax error at
    `ct_commit_nat' expecting action.

    error parsing match "flags.from_ctrl && udp.src == 53": Syntax error at
    `flags.from_ctrl' expecting field name.

  This prevents the affected flows from being installed and can disrupt
  network traffic during the upgrade.

  [Test Plan]

  Deploy a multinode Jammy Yoga OpenStack cloud using updated OVN 22.03 packages.
  Configure instances on different compute hosts, security groups, floating
  IPs, OVN DNS records.

  Confirm that the Southbound database contains flows referencing both
  ct_commit_nat and flags.from_ctrl.

  Verify that:

  * All relevant units install the candidate package and their OVN
    services remain healthy.
  * No ct_commit_nat or flags.from_ctrl parsing errors appear in the
    ovn-controller logs during the upgrade.
  * Controllers converge and process subsequent network configuration.
  * Existing and newly created instances have working east-west and
    floating-IP connectivity.
  * OVN DNS resolution works with security groups enabled.
  * Related ICMP traffic through an OVN load balancer is delivered with
    the correct NAT translation.

  Run the OpenStack functional and upgrade tests and attach the
  results, package versions, and relevant logs before marking verification
  complete.

  [Where problems could occur]

  The patches change logical action parsing, load-balancer handling of
  related traffic, and the ACL treatment of OVN-generated DNS responses.

  Regressions could affect connection tracking or NAT for related ICMP
  traffic, prevent logical flows from being installed, or incorrectly allow
  or block DNS traffic.

  Validation must cover both existing connections and new network
  configuration after the upgrade. DNS testing must confirm that legitimate
  OVN-generated replies work while workload-generated traffic remains
  subject to the configured security-group rules.

  [Other Info]
  After discussion with other Cloud Archive admin, we've deemed that a multi node deployment testing yoga -> zed -> antelope -> bobcat upgrade test is sufficient to validate these patches, and does not require a full charmed openstack deployment. Updated the original testplan, and keep this comment for history.

  --- Original Description

  The charmed OpenStack product currently make use of a step by step
  upgrade process.

  This includes stepping through interim releases of OVN.

  While OVN itself supports direct upgrades between LTS versions, it is
  impractical in the current organization of apt repositories.

  Between 22.03 and 22.04 releases, two new actions has been introduced
  and backported to resolve critical issues.

  These patches have made their way into the ovn 22.03 package in Jammy,
  but not to the intermediate versions currently maintained in UCA.  As
  a consequence, when users attempt to upgrade they will hit an data
  path impacting issue, and find messages below logged by the ovn-
  controller:

  The required patches to resolve this are:
  474bdfcad038 ("Skip only OVN DNS responder packets from OUT_ACL.")
  650f06b9f3e2 ("northd: Allow related traffic through LB")
  31196346fcad ("actions: Add new action called ct_commit_nat")

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2115328/+subscriptions