vmxnet3 1.3.1.0-5 and Kernel 3.13.11.10

[Stefan Bader]

On 26.11.2014 17:00, Jason Strongman wrote:
> Some background.
> 
> I am currently running some performance tests against apache traffic server.
> My initial tests against trusty(latest mainline64bit kernel) was less than
> desired. I then ran the same tests against precise with the latest mainline
> 64bit kernel and performance was much better, but numbers were still not in
> the same range as our RHEL 6.5 build.  Keep in mind all VMs have the same
> virtual hardware profile and same resource allocation.

Just to clarify because I think we use the term mainline differently. You meant
the latest kernel you get when updating (from the Ubuntu archive).

> 
> Since the default precise install gave the best performance, I decided to
> start tweaking this install and go from there.
> 
> On the precise instance I used the kernel referenced in the vmware respoitory
> - 3.2.0-29-virtual(precise)
> 
> I then installed the below module.
> 
> http://packages.vmware.com/tools/esx/latest/ubuntu/dists/precise/main/binary-amd64/vmware-tools-vmxnet3-modules-source_1.3.1.0-5.precise_amd64.deb
>
> 
> 
> This combination of 3.2.0-29-virtual(precise) plus vmxnet3 1.3.1.0-5 resulted
> in the BEST throughput than any other combination. It is a night and day
> difference.  However it's still a bit worse than our RHEL 6.5 build, but not
> bad. We are talking a few milliseconds difference.  So I am thinking the
> major bottleneck is the upstream vmxnet3 module. This is the reason for
> trying to compile the latest external vmxnet3 module against the latest
> mainline kernel.
> 
> So in summary I have tried:
> 
> 1. Trusty  with the latest mainline kernel combined with the native/upstream
> vmxnet3 kernel module. - worst performance 2. Precise with the latest
> mainline kernel combined with the native/upstream vmxnet3 kernel module -
> slightly better performance 3. Precise with the  3.2.0-29-virtual kernel
> combined with the latest vmware provided external vmxnet3 kernel module -
> BEST performance

The way this looks is that vmware themselves considered the upstream kernel
module good enough somewhere in between 3.2 and 3.13 (since they do not provide
a newer version of the external package) and the one for Precise unlikely has
code to be compatible with any kernel changes that came later.

I quickly scanned over changes to vmxnet3 between 3.13 and 3.18-rc3. Nothing
that obviously says fixing performance. One about fixing MSI/MSI-X enablement
made me wonder whether this could be related. Maybe you would see a difference
in /proc/interrupts on instances running Precise with the external module and
one that runs Trusty with the kernel module.

I don't think there will be newer external modules. So the only way forward will
be fixing the in-kernel module. One thing I would try to take the Trusty
instance and try more recent kernels (that MSI/MSI-X change for example came in
3.15). The simplest way for that would be what we usually mean when mentioning
mainline kernels (basically snapshots of upstream kernel without any distro
specific patches but using the config options we would use, where possible).
You would find those at: http://kernel.ubuntu.com/~kernel-ppa/mainline/

I would probably try

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.16.7-utopic/

first. Depending on whether that improves the performance you could go forward
or backwards. That way it is possible to narrow down the range of versions to
look for a fix. Well, of course if there has been a fix...

-Stefan
> 
> 
> 
> 
> 
> On Wed, Nov 26, 2014 at 3:03 AM, Stefan Bader <stefan.bader at canonical.com>
> wrote:
>> 
>> On 26.11.2014 03:45, Jason Strongman wrote:
>>> i am having issues compiling the latest vmxnet3 driver against any of
>>> the 3.13.x kernels. are there any patches i should be using to get this
>>> module to compile? i can compile with no problems against any of the
>>> 3.2.x kernels.
>>> 
>> 
>> Do you really need the external vmxnet3 module? VMWare upstreamed their
>> drivers and 3.13 comes with that driver already. Since you say 3.2
>> compiles, is this a Precise (12.04) base install with the 3.13 HWE kernel?
>> 
>> -Stefan
>> 
>> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20141126/f3c705bb/attachment.sig>