[Bug 1940043] Re: Upgrade from OVN 20.03 to newer OVN version will cause data plane outage
Launchpad Bug Tracker
1940043 at bugs.launchpad.net
Wed Mar 1 00:59:36 UTC 2023
This bug was fixed in the package ovn - 20.03.2-0ubuntu0.20.04.4
---------------
ovn (20.03.2-0ubuntu0.20.04.4) focal; urgency=medium
* Adapt to changes made in previous OVS point release (LP: #1980213):
- d/control: Update required openvswitch build requirement.
- d/p/lp-1980213-treewide-bump-ovs-and-fix-problematic-loops.patch
* Fix upgrade from OVN 20.03 to newer OVN versions (LP: #1940043):
- d/ovn-host.ovn-controller.service: Pass --restart option when
calling `ovn-ctl stop_controller`
- d/p/lp-1940043-0001-Provide-the-option-to-pin-ovn-controller-and-ovn-nor.patch
- d/p/lp-1940043-0002-controller-Allow-pinctrl-thread-to-handle-packet-ins.patch
* d/rules, d/testlist.py, d/flaky-tests.txt:
- Dynamically build list of tests to run from list of test descriptions.
-- Frode Nordahl <frode.nordahl at canonical.com> Wed, 20 Jul 2022
11:42:49 +0200
** Changed in: ovn (Ubuntu Focal)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1940043
Title:
Upgrade from OVN 20.03 to newer OVN version will cause data plane
outage
Status in charm-layer-ovn:
Fix Released
Status in charm-ovn-chassis:
Fix Released
Status in charm-ovn-dedicated-chassis:
Fix Released
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive wallaby series:
Triaged
Status in ovn package in Ubuntu:
Fix Released
Status in ovn source package in Focal:
Fix Released
Status in ovn source package in Hirsute:
Won't Fix
Status in ovn source package in Impish:
Fix Released
Bug description:
[Impact]
When upgrading from OVN 20.03, as made available in Ubuntu Focal, to a newer version of OVN, it is currently not possible to upgrade without causing a data plane outage.
If the user attempts to upgrade the central components first, the ovn-
controller will tear down connectivity to running instances as it may
not fully understand the data structure of a newer database.
If the user attempts to upgrade the ovn-controler first, recent
releases are not guaranteed to understand the older database and
connectivity may remain down until all hypervisors and central
components have been upgraded.
If the user attempts to manually stop the ovn-controller during the
upgrade to avoid it inadvertently tearing down connectivity on central
component upgrade, cloud instances will be deprived of vital services
such as DNS lookup and DHCP.
To fix this situation two changes are needed:
1) Backport of a upstream feature [0] that allows the ovn-controller to detect version mismatch and subsequently refrain from making further changes to the local Open vSwitch instance until the version mismatch is corrected.
2) Make ovn-controller not clear out runtime flow state in Open
vSwitch on exit by updating the ovn-controller systemd service to pass
the `--restart` argument when stopping the controller. This flag
tells the ovn-controller process that it should not clear out Open
vSwitch flows and OVN SB database records on exit, which allows
already installed state to continue operation until the new instance
of the ovn-controller process starts. [1][2][3]
It does not mean that the service will be restarted as opposed to
being stopped, as one might think based on the name of the argument.
This change serves two purposes:
2a) Allow upgrading the ovn-controller to a newer version than the
central components, while retaining connectivity to running instances
until the central components are upgraded.
2b) Minimize the downtime on package upgrade.
[Test Plan]
1. Deploy OpenStack Ussuri from the Focal archive.
2. Launch and instance and confirm connectivity.
3. Add UCA or other PPA with a newer version of OVN and perform upgrade of the OVN components on relevant units in the deployment.
4. Confirm how new version of central components make the ovn-controller log version mismatch as well as show continued connectivity to the test instance.
5. Upgrade data plane units and confirm how the version mismatch situation is resolved and at the same time instances retain connectivity with minimal downtime during the upgrade.
[Regression Potential]
The backported feature is optional and enabled by specifically
entering a key-value pair into the local Open vSwitch database to
enable it. It has also been available upstream for several releases.
The change to the ovn-controller systemd service has been in Ubuntu
since Impish [3] and we have had no reports of side effects of this
change.
[Original Bug Description]
The upstream recommendation for upgrades of OVN is to first upgrade the data plane components (chassis aka. ovn-controller), and then upgrade the central components (the database schema and ovn-northd). The rationale for this is that the new version of the ovn-controller is required to cope with any changes to database schema or how northd programs flows.
However, during the course of rapid OVN development there has also
been introduced changes that make the new ovn-controller not cope with
a old database schema, breaking the recommended upgrade procedure.
To cope with this upstream has introduced a new optional configuration
for the ovn-controller that allows it to detect version
inconsistencies, and when they are present stop it from making changes
to the data plane until the version inconsistency is resolved [0].
For the above mentioned configuration to be effective we also need the
package to call ``ovn-ctl stop_controller`` with the --restart option
so that the ovn-controller does not flush the installed flows on exit.
We should make required changes to packages and charms to allow
upgrades to progress with less data plane outage.
0: https://github.com/ovn-org/ovn/commit/1dd27ea7aea40122c1edbff845e14abaa70c0413
1: https://github.com/ovn-org/ovn/commit/f508fcc14abfaaa13e9f1bf3b5b6bac59bd27a5f
2: https://github.com/ovn-org/ovn/commit/45c7a85dc7f2af56191a47f1357d16b8af618e20
3: https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/ovn/commit/debian/ovn-host.ovn-controller.service?id=3c601ecc13724d3f13ec0cc989f6ffd838f787f8
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-layer-ovn/+bug/1940043/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list