[Bug 1906280] Re: [SRU] Add support for disabling mlockall() calls in ovs-vswitchd

Wed Jan 27 18:52:27 UTC 2021

** Tags added: cdo-release-blocker

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1906280

Title:
  [SRU] Add support for disabling mlockall() calls in ovs-vswitchd

Status in OpenStack neutron-openvswitch charm:
  Fix Committed
Status in charm-ovn-chassis:
  Fix Committed
Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive queens series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Released
Status in openvswitch package in Ubuntu:
  Fix Released
Status in openvswitch source package in Bionic:
  Fix Released
Status in openvswitch source package in Focal:
  Fix Released
Status in openvswitch source package in Groovy:
  Fix Released
Status in openvswitch source package in Hirsute:
  Fix Released

Bug description:
  [Impact]

  Recent changes to systemd rlimit are resulting in memory exhaustion
  with ovs-vswitchd's use of mlockall(). mlockall() can be disabled via
  /etc/defaults/openvswitch-vswitch, however there is currently a bug in
  the shipped ovs-vswitchd systemd unit file that prevents it. The
  package will be fixed in this SRU. Additionally the neutron-
  openvswitch charm will be updated to enable disabling of mlockall()
  use in ovs-vswitchd via a config option.

  More details on the above summary can be found in the following comments:
  https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1906280/comments/16
  https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1906280/comments/19

  ====  Original bug details ===
  Original bug title:

  Charm stuck waiting for ovsdb 'no key "ovn-remote" in Open_vSwitch
  record'

  Original bug details:

  As seen during this Focal Ussuri test run: https://solutions.qa.canonical.com/testruns/testRun/5f7ad510-f57e-40ce-beb7-5f39800fa5f0
  Crashdump here: https://oil-jenkins.canonical.com/artifacts/5f7ad510-f57e-40ce-beb7-5f39800fa5f0/generated/generated/openstack/juju-crashdump-openstack-2020-11-28-03.40.36.tar.gz

  Full history of occurrences can be found here:
  https://solutions.qa.canonical.com/bugs/bugs/bug/1906280

  Octavia's ovn-chassis units are stuck waiting:

  octavia/0                             blocked   idle       1/lxd/8   10.244.8.170    9876/tcp           Awaiting leader to create required resources
    hacluster-octavia/1                 active    idle                 10.244.8.170                       Unit is ready and clustered
    logrotated/63                       active    idle                 10.244.8.170                       Unit is ready.
    octavia-ovn-chassis/1               waiting   executing            10.244.8.170                       'ovsdb' incomplete
    public-policy-routing/45            active    idle                 10.244.8.170                       Unit is ready

  When the db is reporting healthy:

  ovn-central/0*                        active    idle       1/lxd/9   10.246.64.225   6641/tcp,6642/tcp  Unit is ready (leader: ovnnb_db, ovnsb_db)
    logrotated/19                       active    idle                 10.246.64.225                      Unit is ready.
  ovn-central/1                         active    idle       3/lxd/9   10.246.64.250   6641/tcp,6642/tcp  Unit is ready (northd: active)
    logrotated/27                       active    idle                 10.246.64.250                      Unit is ready.
  ovn-central/2                         active    idle       5/lxd/9   10.246.65.21    6641/tcp,6642/tcp  Unit is ready
    logrotated/52                       active    idle                 10.246.65.21                       Unit is ready.

  Warning in the juju unit logs indicates that the charm is blocking on
  a missing key in the ovsdb:

  2020-11-27 23:36:57 INFO juju-log ovsdb:195: Invoking reactive handler: hooks/relations/ovsdb-subordinate/provides.py:97:joined:ovsdb-subordinate
  2020-11-27 23:36:57 DEBUG jujuc server.go:211 running hook tool "relation-get"
  2020-11-27 23:36:57 WARNING ovsdb-relation-changed ovs-vsctl: no key "ovn-remote" in Open_vSwitch record "." column external_ids
  2020-11-27 23:36:57 DEBUG jujuc server.go:211 running hook tool "juju-log"
  2020-11-27 23:36:57 INFO juju-log ovsdb:195: Invoking reactive handler: hooks/relations/ovsdb/requires.py:34:joined:ovsdb
  ==============================

  [Test Case]
  Note: Bionic requires additional testing due to pairing with other SRUS.

  The easiest way to test this is to deploy openstack with the neutron-
  openvswitch charm, using the new charm updates. Once deployed, edit
  /usr/share/openvswitch/scripts/ovs-ctl with an echo to show what
  MLOCKALL is set to. Then toggle the charm config option [1] and look
  at journalctl -xe to find the echo output, which should correspond to
  the mlockall setting.

  [1]
  juju config neutron-openvswitch disable-mlockall=true
  juju config neutron-openvswitch disable-mlockall=false

  [Regression Potential]
  There's potential that this will break users who have come to depend on the incorrect EnvironmentFile setting and environment variable in the systemd unit file for ovs-vswitchd. If that is the case they must be running with modified systemd unit files anyway so it is probably a moot point.

  [Discussion]
  == Groovy ==
  Update (16-12-2020): I chatted briefly with Christian and it sounds like the ltmain-whole-archive.diff may be optional, so I've dropped it from this upload. There are now 2 openvswitch's in the groovy unapproved queue. Please reject the upload from 15-12-2020 and consider accepting the upload from 16-12-2020.
  I have a query out to James and Christian about an undocumented commit that is getting picked up in the groovy upload. It is committed to the ubuntu/groovy branch of the package Vcs. See debian/ltmain-whole-archive.diff and debian/rules in the upload debdiff at http://launchpadlibrarian.net/511453613/openvswitch_2.13.1-0ubuntu1_2.13.1-0ubuntu1.1.diff.gz

  == Bionic ==
  The bionic upload is paired with the following SRUs which will also require verification:
  https://bugs.launchpad.net/bugs/1823295
  https://bugs.launchpad.net/bugs/1881077

  == Package details ==
  New package versions are in progress and can be found at:
  hirsute: https://launchpad.net/ubuntu/+source/openvswitch/2.14.0-0ubuntu2
  groovy: https://launchpad.net/ubuntu/groovy/+queue?queue_state=1&queue_text=openvswitch
  focal: https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=openvswitch
  train: https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/train-staging/+packages?field.name_filter=openvswitch&field.status_filter=published&field.series_filter=
  stein: https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/stein-staging/+packages?field.name_filter=openvswitch&field.status_filter=published&field.series_filter=
  bionic: https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=openvswitch

  == Charm update ==
  https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/767212

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1906280/+subscriptions