[Bug 1720126] Re: [ip link] Message truncated error for large number of passthrough VFs

[Nish Aravamudan]

On 20.10.2017 [07:09:00 -0000], Jan Gutter wrote:
> I concur with option 2), unnecessary deviation will just cause
> confusion.

Thank you for confirming that!

> Regarding the other buffer sizes, the last time I looked they were
> mostly OK. The issue reared its head in this particular case because the
> netlink message that previously had a pretty constant per-netdev
> response size suddenly had the ability to balloon with "no warning". A
> number of workarounds exist (i.e. you have to explicitly ask for the VF
> info), but, in this case we actually want the VF info and iproute2 was
> just unprepared for the size of it.

Ok, that's good to hear.

> I guess the core issue is that it's entirely possible for the kernel to
> add extra netlink attributes to any query response, iproute2 makes the
> assumption that the queries it's making is not necessarily going to
> explode with gigabytes of new annotations and 16k will easily fit any
> current real-world system. A pragmatic approach would probably be to
> handle the "Message Truncated" path with a dynamically sized buffer as
> an exceptional case.

Yep, I can see how iproute2 itself has to move in lockstep with the
kernel, which also means older iproute2 that can run on newer kernels
needs periodic updates like this one.

> Any fix in iproute2 that "properly" addresses this issue has to be
> carefully vetted. Who knows how many inherent races will get exposed if
> the ip command doubles in execution time.

Yep :) I'm fine with eventually doubling the buffer again statically if
that is the conclusion upstream reaches. My guess is that is the
simplest solution.

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1720126

Title:
  [ip link] Message truncated error for large number of passthrough VFs

Status in iproute2 package in Ubuntu:
  Fix Released
Status in iproute2 source package in Trusty:
  New
Status in iproute2 source package in Xenial:
  Confirmed
Status in iproute2 source package in Zesty:
  Fix Released
Status in iproute2 package in CentOS:
  Unknown

Bug description:
  [Impact]

  When querying a Physical Function netdev with a large amount of VF's
  (more than 30), the resulting return message can overflow the 16K
  netlink message buffer.

  This can be fixed by enabling message peeking on the socket and
  resizing the buffer on receive, or by simply enlarging the receive
  buffer.

  Since there's an upper limit to the number of VF's per PF, it's
  relatively sane to just enlarge the receive buffer. Please see the
  attached patch.

  [Test Case]

  # Set up 60 VF's on an SR-IOV device
  ip link show > /dev/null

  Observe the following:
  Message truncated
  Message truncated
  Message truncated

  [Regression Potential]

  1) Applications relying on the broken behaviour will need to be updated, but it would be a really dubious use case.
  2) Increasing the rx buffer size increases the memory footprint (but realistically, this is tiny).
  3) Extra processing time is now needed to parse the larger buffer, in the case that a call to "ip link" is on the critical time path of an application, (called multiple times in a tight loop, for example), it would affect load.

  [Other Info]

  Observed on Ubuntu kernel 4.4.0-93-generic on both 14.04 and 16.04

  =====================================================================================================
  Ubuntu16 system

  stack at cluster04:~$ lsb_release -a
  No LSB modules are available.
  Distributor ID:	Ubuntu
  Description:	Ubuntu 16.04.3 LTS
  Release:	16.04
  Codename:	xenial

  stack at cluster04:~$ uname -r
  4.4.0-93-generic

  stack at cluster04:~$ apt-cache policy iproute2
  iproute2:
    Installed: 4.3.0-1ubuntu3.16.04.1
  Version table:
  *** 4.3.0-1ubuntu3.16.04.1 500
          500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
  =================================================================================================

  Ubuntu14 system:
  root at boomslang:~# lsb_release -a
  No LSB modules are available.
  Distributor ID:	Ubuntu
  Description:	Ubuntu 14.04.3 LTS
  Release:	14.04
  Codename:	trusty

  root at boomslang:~# uname -r
  4.4.0-96-generic

  root at boomslang:~# apt-cache policy iproute2
  iproute2:
    Installed: 3.12.0-2ubuntu1
    Version table:
   *** 3.12.0-2ubuntu1 0
          500 http://za.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1720126/+subscriptions