[Bug 1737866] Re: Too many open files when large number of routers on a host

[Hua Zhang]

Failed to verify trusty-proposed/mitaka/main
2.5.5-0ubuntu0.16.04.1~cloud0 since trusty doesn't use systemd as
default

root at trusty:/tmp/openvswitch-2.5.5# grep -r '1048576' ./debian/
./debian/changelog:  * Bump nofiles to 1048576 for ovs daemons (LP: #1737866).
./debian/openvswitch-switch.openvswitch-nonetwork.service:LimitNOFILE=1048576

root at trusty:~#  grep -r '1048576' /lib/systemd/system/openvswitch*
/lib/systemd/system/openvswitch-nonetwork.service:LimitNOFILE=1048576

root at trusty:~# grep -r 'Max open files' /proc/`pidof ovsdb-server`/limits
Max open files            1024                 4096                 files

root at trusty:~# grep -r 'Max open files' /proc/`pidof ovs-vswitchd`/limits
Max open files            65535                65535                files

** Tags removed: verification-mitaka-needed
** Tags added: verification-mitaka-failed

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1737866

Title:
  Too many open files when large number of routers on a host

Status in OpenStack neutron-openvswitch charm:
  Invalid
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  Fix Released
Status in Ubuntu Cloud Archive ocata series:
  Fix Released
Status in Ubuntu Cloud Archive pike series:
  Fix Released
Status in openvswitch package in Ubuntu:
  Fix Released
Status in openvswitch source package in Xenial:
  Fix Released
Status in openvswitch source package in Artful:
  Won't Fix
Status in openvswitch source package in Bionic:
  Fix Released

Bug description:
  [Impact]
  OpenStack environments running large numbers of routers and dhcp agents on a single host can hit the NOFILES limit in OVS, resulting in broken operation of virtual networking.

  [Test Case]
  Deploy openstack environment; create large number of virtual networks and routers.
  OVS will start to error with 'Too many open files'

  [Regression Potential]
  Minimal - we're just increasing the NOFILE limit via the systemd service definition.

  [Original Bug Report]
  When there are a large number of routers and dhcp agents on a host, we see a syslog error repeated:

  "hostname ovs-vswitchd: ovs|1762125|netlink_socket|ERR|fcntl: Too many
  open files"

  If I check the number of filehandles owned by the pid for "ovs-
  vswitchd unix:/var/run/openvswitch/db.sock" I see close to/at 65535
  files.

  If I then run the following, we double the limit and (in our case) saw
  the count rise to >80000:

  prlimit -p $pid --nofile=131070

  We need to be able to:
  - monitor via nrpe, if the process is running short on filehandles
  - configure the limit so we have the option to not run out.

  Currently, if I restart the process, we'll lose this setting.

  Needless to say, openvswitch running out of filehandles causes all
  manner of problems for services which use it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1737866/+subscriptions