[Bug 1818614] Please test proposed package

[Timo Aaltonen]

Hello Slawek, or anyone else affected,

Accepted neutron into cosmic-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/neutron/2:13.0.2-0ubuntu3 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-cosmic to verification-done-cosmic. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-cosmic. In either case, without details of
your testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1818614

Title:
  [SRU] Various L3HA functional tests fails often

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive pike series:
  Fix Committed
Status in Ubuntu Cloud Archive queens series:
  Triaged
Status in Ubuntu Cloud Archive rocky series:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Committed
Status in neutron source package in Cosmic:
  Fix Committed
Status in neutron source package in Disco:
  Fix Released

Bug description:
  [Impact]
  Need to get this added to the Ubuntu packages in order to safeguard against missed VRRP transitions due to ip -o monitor not running at the time the transition occurs. We have seen many cases in the fields where neutron routers end up as active on multiple l3 agents (via neutron api) which leads to a number of problems.

  [Test Case]
  * deploy Openstack (any version that supports l3ha)
  * create HA router with max-l3-agents=2
  * check neutron l3-agent-list-hosting-router for master location
  * on both hosts that are running the l3-agent do

  pid=`pgrep -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"`
  ps -f --ppid $pid
  pkill -f "/usr/bin/neutron-keepalived-state-change --router_id=$ROUTER_UUID"
  ps -f --ppid $pid <<<<<<<<<<< this should return nothing now
  pkill -f "/var/lib/neutron/ha_confs/$ROUTER_UUID/keepalived.conf"

  * without this patch you should now see both agents reporting the router as "active"
  * with the patch this should not happen (once neutron-keepalived-state-change has been restarted)

  [Regression Potential]
  These patches have already landed in corresponding upstream branches and therefore have undergone reviews + unit and functional testing upstream, therefore regression potential is expected to be low.

  ====================================================================

  Recently many L3 HA related functional tests are failing.
  The common thing in all those errors is fact that it fails when waiting for l3 ha router to become master.

  Example stack trace:

  ft2.12: neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase.test_ha_router_lifecycle_StringException: Traceback (most recent call last):
    File "neutron/tests/base.py", line 174, in func
      return f(self, *args, **kwargs)
    File "neutron/tests/base.py", line 174, in func
      return f(self, *args, **kwargs)
    File "neutron/tests/functional/agent/l3/test_ha_router.py", line 81, in test_ha_router_lifecycle
      self._router_lifecycle(enable_ha=True, router_info=router_info)
    File "neutron/tests/functional/agent/l3/framework.py", line 274, in _router_lifecycle
      common_utils.wait_until_true(lambda: router.ha_state == 'master')
    File "neutron/common/utils.py", line 690, in wait_until_true
      raise WaitTimeout(_("Timed out after %d seconds") % timeout)
  neutron.common.utils.WaitTimeout: Timed out after 60 seconds

  Example failure: http://logs.openstack.org/79/633979/21/check/neutron-
  functional-python27/ce7ef07/logs/testr_results.html.gz

  Logstash query:
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ha_state%20%3D%3D%20'master')%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1818614/+subscriptions