Xenial - OpenvSwitch with DPDK binding to 10G NIC, not working...

Wed Mar 9 01:49:27 UTC 2016

On 8 March 2016 at 20:59, Martinx - ジェームズ <thiagocmartinsc at gmail.com> wrote:

> Hey Christian,
>
> Thank you for your hard work!
>
> Replying inline below...
>
> On 8 March 2016 at 04:48, Christian Ehrhardt <
> christian.ehrhardt at canonical.com> wrote:
>
>> Hi Thiago,
>> comments inline below ...
>>
>> Christian Ehrhardt
>> Software Engineer, Ubuntu Server
>> Canonical Ltd
>>
>> From: Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>> Date: Mon, Mar 7, 2016 at 2:28 PM
>> Subject: Re: Xenial - OpenvSwitch with DPDK binding to 10G NIC, not
>> working...
>> To: Ubuntu Core developers <ubuntu-devel-discuss at lists.ubuntu.com>
>>
>>
>> On 7 March 2016 at 16:40, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>> wrote:
>>
>>> Hey guys,
>>>
>>>  I'm trying to create two OVS bridges powered by DPDK, on Xenial, but
>>> I'm facing problems...
>>>
>>>  First, there is no documentation about how to do this... So far, I
>>> found the following:
>>>
>>>
>>>  - This link doesn't how to bind real NIC:
>>>
>>> https://software.intel.com/en-us/articles/using-open-vswitch-with-dpdk-on-ubuntu
>>>
>>>  - This one isn't specific to Ubuntu OVS + DPDK packages:
>>>  https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md
>>>
>>>  - Similar with above but, very old... Also not related to Ubuntu
>>> packages:
>>>  https://feiskyer.github.io/2015/03/01/open-vswitch-over-dpdk-on-ubuntu/
>>>
>>>
>> Yeah docs are often outdated, I have realized that as well.
>> Not because people are lazy, but because the environment is changing so
>> fast.
>> It is already planned to do a blog post or wiki or such along 16.04
>> release to guide users of the very updated dpdk / openvsiwtch-dpdk.
>> But for now we are working on so many issues that things are changing too
>> fast to pre-write that document.
>>
>> To add a bit I also found this one useful:
>>
>> http://www.ran-lifshitz.com/2015/08/28/open-vswitch-netdev-dpdk-with-vhost-user-support-update/
>> But be aware of
>> https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1546565
>> if you want to pass it to guests.
>> At least there the log entries will be pretty obvious with "permission
>> denied" on the vhost-user sockets.
>>
>
> Yes, I totally understand that. I want to be ready for 16.04 release,
> since day one. Currently, my company works with CentOS and manually
> installed DPDK, which is just plain terrible, and I am pushing Ubuntu!
>
> I'm not facing bug #1546565, I'm not there yet.
>
>
>>
>>
>>
>>>  So, to begin with, I start with Intel.com guide, it worked! But without
>>> real NIC.
>>>
>>>  Then, I tried to bind it to real NIC, like this:
>>>
>>>  Added two NIC PCI id to "/etc/dpdk/interfaces" file:
>>>
>>> ---
>>> pci     0000:01:00.0    uio-pci-generic
>>> pci     0000:01:00.1    uio-pci-generic
>>> ---
>>>
>>>  Then, reboot, I can see both NICs using "DPDK-compatible driver" by
>>> running: "dpdk_nic_bind --status".
>>>
>>>  Trying to create new OVS Bridges with DPDK devices, failing:
>>>
>>> ---
>>> root at xenial-1:~# ovs-vsctl add-br ovsbr0
>>> root at xenial-1:~# ovs-vsctl set bridge ovsbr0 datapath_type=netdev
>>> root at xenial-1:~# ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0
>>> type=dpdk
>>> ovs-vsctl: Error detected while setting up 'dpdk0'.  See ovs-vswitchd
>>> log for details.
>>>
>> [...]
>>
>>> 2016-03-07T19:34:07.532Z|00136|bridge|WARN|could not open network device
>>> dpdk0 (No such device)
>>>
>> [...]
>>
>>>         Port "dpdk0"
>>>             Interface "dpdk0"
>>>                 type: dpdk
>>>                 error: "could not open network device dpdk0 (No such
>>> device)"
>>>
>>
>> I had seen something similar.
>> The particular one you are describing here was for me a device that was
>> either no bound by dpdk or had no compatible driver.
>> But according to what I see both should be ok for you.
>>
>
> Yes, I have DPDK compatible 10G NIC cards, I can bind to both uio-generic
> and vfio-pci but, OpenvSwitch+DPDK doesn't find it.
>
> The following command always fails:
>
> ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk
>
>
>> Often I realized that the vswitch log alone isn't too helpful.
>> I recommend using journalctl along to get the EAL messages of dpdk as
>> well.
>>
>
> Sure, I agree.
>
>
>>
>> To help you I created a merged log (time goes top down) which has both
>> logs and the command I used for a successful ovs-dpdk port add.
>> I think that will help you to see where your systems behaviour starts to
>> differ.
>>
>> => http://paste.ubuntu.com/15326358/
>>
>> In your case you might e.g. see "skipped device" there leading eventually
>> to your dpdk0 not found issue.
>>
>> Also you might already have done that, but just not mentioned. Still to
>> help as much as possible - have you set DPDK_OPTS, I for example did that
>> before (re)starting it?
>> For example
>>     echo "set DPDK_OPTS='--dpdk -c 0x1 -n 4 -m 4096'" >>
>> /etc/default/openvswitch-switch
>>
>
> It is good to see that it is working for you! Maybe I am almost there...
>
> Yes, I tried all of this (specially INSTALL.DPDK.md
> <https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md>)...
>
> I can see that [ps xufawww] OVS is running with --dpdk options.
>
> I'll try now, to follow your pastebin step by step...
>
>
>>
>>
>> [...]
>>
>> I have also seen the part about vfio-pci in your mail and thank you for
>> reporting the bug, I'll handle work and communication of this issue in the
>> bug.
>>
>
> Awesome! I'm see that you already fixed it! That was fast!   :-D
>
> Thank you again!
>
> Cheers!
> Thiago
>

Yeah, still doesn't work... I am unable to reproduce what you did here:
http://paste.ubuntu.com/15326358/

I'm seeing the problem over and over, no matter which driver I'm
using uio-pci-generic or vfio-pci...

I'll wait for instructions, no problem... Neither Intel guys knows how to
do this on Ubuntu, so, who am I?   :-P

My post on Intel forum is now available (moderated):

https://software.intel.com/en-us/forums/networking/topic/611450

Cheers man!

Thanks,
Thiago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-devel-discuss/attachments/20160308/a1207075/attachment.html>