[Bug 1907081] Re: Clustered OVN database is not upgraded on package upgrade

[Frode Nordahl]

** Description changed:

  [Impact]
  On upgrade of the OVN packages it may be necessary to perform a upgrade to the Northbound and Southbound databases.
  
  Failure to do so may lead to loss of connectivity between participating
  nodes as the software components will attempt to make use of columns
  that are not available in the database.
  
  The upgrade process has been performed automatically by the upstream
  init script by default since inception, both for a local and clustered
  setup. But as discussed below recent changes has inadvertently omitted
- this behavior.
+ this behavior for clustered databases.
  
  [Test Case]
  
- Non-clustered scenario:
- Install the ovn-central components in a container using the in-release focal package and start the database and ovn-northd services.
+ Non-clustered scenario as reference test:
+ Install the ovn-central package in a container using the in-release focal package and start the database and ovn-northd services.
  
  Upgrade the container to the OVN packages from in-release Groovy and
- observe the ovn-northd service complaining about missing columns in the
+ observe the package performing the database upgrade and subsequently
+ ovn-northd service not complaining about missing columns in the
  database.
- 
- Repeat the test with the proposed packages and observe that the
- databases being upgraded on restart of services after package upgrade.
  
  Clustered scenario:
  Install the ovn-central charm across three containers and necessary dependencies. Perform package upgrade as outlined above and compare how in-relase and proposed packages behave.
  
  [Regression Potential]
  As we are restoring the intended behavior the regression potential is minimal.
  
  [Original Bug Report]
  In the systemd service we make use of the `ovn-ctl` script `run_nb_ovsdb` and `run_sb_ovsdb` sub-commands introduced in [0]. These sub-commands fit nicely with systemd's expectations of modern daemons to no longer detachand run in the background.
  
  However, the change in [0] has the side effect of disabling automatic
  upgrading of clustered databases. Previously this would have been done
  on every startup [1].
  
  A recent commit to master [2] addresses this and uses the combination of
  presence of `--db-*-cluster-local-addr` and non-presence of the `--db
  -*-cluster-remote-addr` to determine if the upgrade should be run.
  
  We should backport [2] to our supported OVN packages to prepare for
  supporting upgrades that require database schema changes. We may also
  need to change the behavior of the ovn-central charm to not set the
  `--db-*-cluster-remote-addr` argument on the leader unit.
  
  0: https://github.com/ovn-org/ovn/commit/6444059b5f9444ce06634794d275257f945a6ce5
  1: https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L312-L314
  2: https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ovn in Ubuntu.
https://bugs.launchpad.net/bugs/1907081

Title:
  Clustered OVN database is not upgraded on package upgrade

Status in charm-ovn-central:
  In Progress
Status in ovn package in Ubuntu:
  In Progress

Bug description:
  [Impact]
  On upgrade of the OVN packages it may be necessary to perform a upgrade to the Northbound and Southbound databases.

  Failure to do so may lead to loss of connectivity between
  participating nodes as the software components will attempt to make
  use of columns that are not available in the database.

  The upgrade process has been performed automatically by the upstream
  init script by default since inception, both for a local and clustered
  setup. But as discussed below recent changes has inadvertently omitted
  this behavior for clustered databases.

  [Test Case]

  Non-clustered scenario as reference test:
  Install the ovn-central package in a container using the in-release focal package and start the database and ovn-northd services.

  Upgrade the container to the OVN packages from in-release Groovy and
  observe the package performing the database upgrade and subsequently
  ovn-northd service not complaining about missing columns in the
  database.

  Clustered scenario:
  Install the ovn-central charm across three containers and necessary dependencies. Perform package upgrade as outlined above and compare how in-relase and proposed packages behave.

  [Regression Potential]
  As we are restoring the intended behavior the regression potential is minimal.

  [Original Bug Report]
  In the systemd service we make use of the `ovn-ctl` script `run_nb_ovsdb` and `run_sb_ovsdb` sub-commands introduced in [0]. These sub-commands fit nicely with systemd's expectations of modern daemons to no longer detachand run in the background.

  However, the change in [0] has the side effect of disabling automatic
  upgrading of clustered databases. Previously this would have been done
  on every startup [1].

  A recent commit to master [2] addresses this and uses the combination
  of presence of `--db-*-cluster-local-addr` and non-presence of the
  `--db-*-cluster-remote-addr` to determine if the upgrade should be
  run.

  We should backport [2] to our supported OVN packages to prepare for
  supporting upgrades that require database schema changes. We may also
  need to change the behavior of the ovn-central charm to not set the
  `--db-*-cluster-remote-addr` argument on the leader unit.

  0: https://github.com/ovn-org/ovn/commit/6444059b5f9444ce06634794d275257f945a6ce5
  1: https://github.com/ovn-org/ovn/blob/5c2d311b8b7b4d5c3a619de72be6a433aa4c44db/utilities/ovn-ctl#L312-L314
  2: https://github.com/ovn-org/ovn/commit/67e2f386cc838d0b0f9b4b5da7fe611e1113b70c

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ovn-central/+bug/1907081/+subscriptions