[Bug 1720126] Re: [ip link] Message truncated error for large number of passthrough VFs

[Jan Gutter]

@nacc

Thanks so much for the explanation. I also found
https://wiki.ubuntu.com/ServerTeam/KnowledgeBase#Merge_Proposals_and_Reviewing
that details a bit more of the internal processes. As relative outsiders
to the Ubuntu process, I'd appreciate it very much if you could handle
that part for Monique's patches. I can be on hand to answer technical
questions if required.

Regarding the buffer size choice, it's very arbitrary as Phil said. I'm
pretty sure we came to the same conclusion independently (libvirt and
libnl had very similar issues) and the workaround is obvious. 32k seems
to work for 64 VF's (our test case), but breaks with 128 VF's. Not a lot
of machines can handle 128 concurrent VF's. I typed 64k "just because".
libvirt+libnl allow message peeking. However, iproute2 uses netlink
directly. So, implementing a similar idea would require an entirely new
receive codepath with all the fun of finding out where new exception
paths occur: something to be done on tip and not suitable for backport
without thorough vetting.

I'm sure it'll save a lot of time once the kinks have been worked out of
the automation, backports are quite the double-edged sword.

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1720126

Title:
  [ip link] Message truncated error for large number of passthrough VFs

Status in iproute2 package in Ubuntu:
  Fix Released
Status in iproute2 source package in Trusty:
  New
Status in iproute2 source package in Xenial:
  Confirmed
Status in iproute2 source package in Zesty:
  Fix Released
Status in iproute2 package in CentOS:
  Unknown

Bug description:
  [Impact]

  When querying a Physical Function netdev with a large amount of VF's
  (more than 30), the resulting return message can overflow the 16K
  netlink message buffer.

  This can be fixed by enabling message peeking on the socket and
  resizing the buffer on receive, or by simply enlarging the receive
  buffer.

  Since there's an upper limit to the number of VF's per PF, it's
  relatively sane to just enlarge the receive buffer. Please see the
  attached patch.

  [Test Case]

  # Set up 60 VF's on an SR-IOV device
  ip link show > /dev/null

  Observe the following:
  Message truncated
  Message truncated
  Message truncated

  [Regression Potential]

  1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case.
  2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny).
  3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load.

  [Other Info]

  Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04

  =====================================================================================================
  Ubuntu16 system

  stack at cluster04:~$ lsb_release -a
  No LSB modules are available.
  Distributor ID:	Ubuntu
  Description:	Ubuntu 16.04.3 LTS
  Release:	16.04
  Codename:	xenial

  stack at cluster04:~$ uname -r
  4.4.0-93-generic

  stack at cluster04:~$ apt-cache policy iproute2
  iproute2:
    Installed: 4.3.0-1ubuntu3.16.04.1
  Version table:
  *** 4.3.0-1ubuntu3.16.04.1 500
          500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
  =================================================================================================

  Ubuntu14 system:
  root at boomslang:~# lsb_release -a
  No LSB modules are available.
  Distributor ID:	Ubuntu
  Description:	Ubuntu 14.04.3 LTS
  Release:	14.04
  Codename:	trusty

  root at boomslang:~# uname -r
  4.4.0-96-generic

  root at boomslang:~# apt-cache policy iproute2
  iproute2:
    Installed: 3.12.0-2ubuntu1
    Version table:
   *** 3.12.0-2ubuntu1 0
          500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+subscriptions