[Bug 1752838] Re: [SRU] neutron-ovs-cleanup runs earlier than ovsdb-server when there are many port

[Corey Bryant]

I reviewed the systemd unit files that have a dependency on neutron-ovs-
cleanup and I think we could use some updates there as well to tighten
up the dependency on neutron-ovs-cleanup.

We currently have the following and I think the latter 3 should be
updated with a Requires like neutron-openvswitch-agent.service has. The
After= only handles order when unit files are started together, whereas
Requires= will start the required service if it is not already active
and will fail this serivce if the required service fails to start.

/lib/systemd/system/neutron-openvswitch-agent.service
-----------------------------------------------------
[Unit]
Description=Openstack Neutron Open vSwitch Plugin Agent
After=mysql.service postgresql.service rabbitmq-server.service keystone.service
Requires=neutron-ovs-cleanup.service

/lib/systemd/system/neutron-l3-agent.service
--------------------------------------------
[Unit]
Description=OpenStack Neutron L3 agent
After=mysql.service postgresql.service rabbitmq-server.service keystone.service openvswitch-switch.service neutron-ovs-cleanup.service

/lib/systemd/system/neutron-dhcp-agent.service
----------------------------------------------
[Unit]
Description=OpenStack Neutron DHCP agent
After=mysql.service postgresql.service rabbitmq-server.service keystone.service neutron-ovs-cleanup.service

/lib/systemd/system/nova-compute.service
----------------------------------------
[Unit]
Description=OpenStack Compute
After=libvirtd.service postgresql.service mysql.service keystone.service rabbitmq-server.service ntp.service neutron-ovs-cleanup.service

Another point of concern was that neutron-ovs-cleanup may not finish prior to dependent units starting. That won't happen with the After= option specified. From the systemd.service man page:
"Note that systemd will consider the unit to be in the state "starting" until the program has terminated, so ordered dependencies will wait for the program to finish before starting themselves."

neutron-ovs-cleanup is a Type=oneshot unit with RemainAfterExit=yes,
which means that it will run

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to neutron in Ubuntu.
https://bugs.launchpad.net/bugs/1752838

Title:
  [SRU] neutron-ovs-cleanup runs earlier than ovsdb-server when there
  are many port

Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive newton series:
  Triaged
Status in Ubuntu Cloud Archive ocata series:
  Triaged
Status in Ubuntu Cloud Archive pike series:
  Triaged
Status in Ubuntu Cloud Archive queens series:
  Triaged
Status in neutron package in Ubuntu:
  Triaged
Status in neutron source package in Xenial:
  Triaged
Status in neutron source package in Artful:
  Triaged
Status in neutron source package in Bionic:
  Triaged

Bug description:
  [Impact]

  In code, Basically all ubuntu version could be affected.
  I'm trying to do test them

  Xenial - Tested, Affected
  Artful -
  Bionic -

  UCA-Mitaka - 
  UCA-Newton -
  UCA-Ocata -
  UCA-Pike -
  UCA-Queens -
  ..?

  neutron-ovs-cleanup service runs earlier than ovsdb-server sometimes.

  then, neutron-ovs-cleanup service's status remains failed
  with below error msg

  "ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-
  vsctl: unix:/var/run/openvswitch/db.sock: database connection failed
  (No such file or directory)"

  that is fine for booting.

  but.

  If someday upgrade pkgs related to neutron, it runs neutron-ovs-
  cleanup again because it is not in active status.

  eventually, neutron-ovs-cleanup deletes all ports on running compute
  nodes.

  related logs here https://paste.ubuntu.com/p/QkkwP2vPDv/

  [Test Case]

  1. deploy openstack env
  2. run below scripts to add many ports
  #!/bin/bash

  for i in {0..1000}
  do
    juju ssh nova-compute/0 "sudo ip link add dummy$i type dummy;sudo ip link set name t-test$i dev dummy$i;sudo ovs-vsctl add-port br-int t-test$i";
  done

  3. systemctl list-units --failed
  - check if there is failed neutron-ovs-cleanup service

  [Regression]
  Changes dependency on debian/neutron-openvswitch-agent.neutron-ovs-cleanup.service.in
  need to reboot if this patch is affected. If this symptom is not affected your env. no need to reboot.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1752838/+subscriptions