[Bug 1737866] Re: Too many open files when large number of routers on a host

[Hua Zhang]

I have verified openvswitch-switch=2.5.5-0ubuntu0.16.04.1, it looks good
to me.

root at 16.04:/tmp/ovs/openvswitch-2.5.5$ grep -r '1048576' ./debian/
./debian/changelog:  * Bump nofiles to 1048576 for ovs daemons (LP: #1737866).
./debian/openvswitch-switch.openvswitch-nonetwork.service:LimitNOFILE=1048576

root at 16.04:~$ grep -r '1048576' /lib/systemd/system/openvswitch*
/lib/systemd/system/openvswitch-nonetwork.service:LimitNOFILE=1048576

root at node1:~# grep -r '1048576' /proc/`pidof ovsdb-server`/limits
Max open files            1048576              1048576              files

root at node1:~# grep -r '1048576' /proc/`pidof ovs-vswitchd`/limits
Max open files            1048576              1048576              files

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1737866

Title:
  Too many open files when large number of routers on a host

Status in OpenStack neutron-openvswitch charm:
  Invalid
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive mitaka series:
  In Progress
Status in Ubuntu Cloud Archive ocata series:
  Fix Committed
Status in Ubuntu Cloud Archive pike series:
  Fix Committed
Status in openvswitch package in Ubuntu:
  Fix Released
Status in openvswitch source package in Xenial:
  Fix Committed
Status in openvswitch source package in Artful:
  Won't Fix
Status in openvswitch source package in Bionic:
  Fix Released

Bug description:
  [Impact]
  OpenStack environments running large numbers of routers and dhcp agents on a single host can hit the NOFILES limit in OVS, resulting in broken operation of virtual networking.

  [Test Case]
  Deploy openstack environment; create large number of virtual networks and routers.
  OVS will start to error with 'Too many open files'

  [Regression Potential]
  Minimal - we're just increasing the NOFILE limit via the systemd service definition.

  [Original Bug Report]
  When there are a large number of routers and dhcp agents on a host, we see a syslog error repeated:

  "hostname ovs-vswitchd: ovs|1762125|netlink_socket|ERR|fcntl: Too many
  open files"

  If I check the number of filehandles owned by the pid for "ovs-
  vswitchd unix:/var/run/openvswitch/db.sock" I see close to/at 65535
  files.

  If I then run the following, we double the limit and (in our case) saw
  the count rise to >80000:

  prlimit -p $pid --nofile=131070

  We need to be able to:
  - monitor via nrpe, if the process is running short on filehandles
  - configure the limit so we have the option to not run out.

  Currently, if I restart the process, we'll lose this setting.

  Needless to say, openvswitch running out of filehandles causes all
  manner of problems for services which use it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1737866/+subscriptions