[Bug 1928179] [NEW] [Victoria] Amphora fails to plug vip

Frode Nordahl 1928179 at bugs.launchpad.net
Wed May 12 07:31:16 UTC 2021 

Public bug reported:

The amphora agent will say:
2021-05-12 06:44:13.767 1318 INFO octavia.amphorae.backends.health_daemon.health_daemon [-] Health Manager Sender starting.
2021-05-12 06:44:22.418 1319 DEBUG octavia.amphorae.backends.agent.api_server.osutils [-] b'' _bring_if_up /usr/lib/python3/dist-packages/octavia/amphorae/backends/agent/api_server/osutils.py:208
2021-05-12 06:44:22.425 1319 DEBUG octavia.amphorae.backends.agent.api_server.osutils [-] b'' _bring_if_up /usr/lib/python3/dist-packages/octavia/amphorae/backends/agent/api_server/osutils.py:212
2021-05-12 06:44:22.445 1319 ERROR octavia.amphorae.backends.agent.api_server.osutils [-] Failed to ifup eth1 due to error: Command '['ip', 'netns', 'exec', 'amphora-haproxy', 'ifup', 'eth1']' returned non-zero exit status 1. b'/bin/sh: 1: /usr/local/bin/udp-masquerade.sh: not found\nifup: failed to bring up eth1\n': subprocess.CalledProcessError: Command '['ip', 'netns', 'exec', 'amphora-haproxy', 'ifup', 'eth1']' returned non-zero exit status 1.

The octavia-worker will say:
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server [-] Exception during message handling: taskflow.exceptions.WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error, Failure: octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error]
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/octavia/controller/queue/v1/endpoints.py", line 45, in create_load_balancer
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     self.worker.create_load_balancer(load_balancer_id, flavor,
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 329, in wrapped_f
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     return self.call(f, *args, **kw)
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 409, in call
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 356, in iter
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     return fut.result()
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     return self.__get_result()
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     raise self._exception
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 412, in call
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/octavia/controller/worker/v1/controller_worker.py", line 370, in create_load_balancer
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     create_lb_tf.run()
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 341, in reraise_if_any
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     raise exc.WrappedFailure(failures)
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server taskflow.exceptions.WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error, Failure: octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error]
2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server 

Reverting to an Ussuri Amphora while stayng with an Victoria control
plane makes it work.

** Affects: snap-octavia-diskimage-retrofit
     Importance: Undecided
         Status: New

** Affects: octavia (Ubuntu)
     Importance: Undecided
         Status: New

** Also affects: snap-octavia-diskimage-retrofit
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to octavia in Ubuntu.
https://bugs.launchpad.net/bugs/1928179

Title:
  [Victoria] Amphora fails to plug vip

Status in snap-octavia-diskimage-retrofit:
  New
Status in octavia package in Ubuntu:
  New

Bug description:
  The amphora agent will say:
  2021-05-12 06:44:13.767 1318 INFO octavia.amphorae.backends.health_daemon.health_daemon [-] Health Manager Sender starting.
  2021-05-12 06:44:22.418 1319 DEBUG octavia.amphorae.backends.agent.api_server.osutils [-] b'' _bring_if_up /usr/lib/python3/dist-packages/octavia/amphorae/backends/agent/api_server/osutils.py:208
  2021-05-12 06:44:22.425 1319 DEBUG octavia.amphorae.backends.agent.api_server.osutils [-] b'' _bring_if_up /usr/lib/python3/dist-packages/octavia/amphorae/backends/agent/api_server/osutils.py:212
  2021-05-12 06:44:22.445 1319 ERROR octavia.amphorae.backends.agent.api_server.osutils [-] Failed to ifup eth1 due to error: Command '['ip', 'netns', 'exec', 'amphora-haproxy', 'ifup', 'eth1']' returned non-zero exit status 1. b'/bin/sh: 1: /usr/local/bin/udp-masquerade.sh: not found\nifup: failed to bring up eth1\n': subprocess.CalledProcessError: Command '['ip', 'netns', 'exec', 'amphora-haproxy', 'ifup', 'eth1']' returned non-zero exit status 1.

  The octavia-worker will say:
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server [-] Exception during message handling: taskflow.exceptions.WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error, Failure: octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error]
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/octavia/controller/queue/v1/endpoints.py", line 45, in create_load_balancer
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     self.worker.create_load_balancer(load_balancer_id, flavor,
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 329, in wrapped_f
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     return self.call(f, *args, **kw)
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 409, in call
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 356, in iter
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     return fut.result()
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     return self.__get_result()
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     raise self._exception
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 412, in call
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/octavia/controller/worker/v1/controller_worker.py", line 370, in create_load_balancer
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     create_lb_tf.run()
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 341, in reraise_if_any
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server     raise exc.WrappedFailure(failures)
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server taskflow.exceptions.WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error, Failure: octavia.amphorae.drivers.haproxy.exceptions.InternalServerError: Internal Server Error]
  2021-05-12 06:44:27.046 129193 ERROR oslo_messaging.rpc.server 

  Reverting to an Ussuri Amphora while stayng with an Victoria control
  plane makes it work.

To manage notifications about this bug go to:
https://bugs.launchpad.net/snap-octavia-diskimage-retrofit/+bug/1928179/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list