Xenial - OpenvSwitch with DPDK binding to 10G NIC, not working...
Martinx - ジェームズ
thiagocmartinsc at gmail.com
Sat Mar 19 05:49:14 UTC 2016
Oh, I forgot to mention that my /etc/dpdk/dpdk.conf have:
---
NR_1G_PAGES=2
---
>From what I'm seeing, my OpenvSwitch-DPDK is not using / seeing PMD.
So, I'm researching about this but, I'll talk with you on Monday...
Have a nice weekend Christian!
Thanks,
Thiago
On 18 March 2016 at 13:16, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
wrote:
> Hello Christian!
>
> Lets make a checklist...
>
>
> 1- Packages "dpdk and openvswitch-switch-dpdk", installed;
>
>
> 2- /etc/default/grub have:
>
> GRUB_CMDLINE_LINUX_DEFAULT="quiet splash default_hugepagesz=1GB
> hugepagesz=1G hugepages=4"
> GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 BOOTIF=01-b0-83-fe-d7-12-fc"
>
> Linux CMD Line:
>
> Command line: BOOT_IMAGE=/vmlinuz-4.4.0-13-generic
> root=/dev/mapper/xenial-1--vg-root ro console=ttyS0,115200n8
> BOOTIF=01-b0-83-fe-d7-12-fc quiet splash default_hugepagesz=1GB
> hugepagesz=1G hugepages=4 vt.handoff=7
>
>
> 3- /etc/dpdk/interfaces have:
>
> pci 0000:01:00.0 uio_pci_generic
> pci 0000:01:00.1 uio_pci_generic
>
>
> 4- /etc/default/openvswitch-switch have:
>
> DPDK_OPTS='--dpdk -c 0x1 -n 4 --socket-mem 1024,0'
>
>
> 5- /etc/alternatives/ovs-vswitchd: symbolic link to
> /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
>
>
> 6- My CPU supports 1G Hugepages (it have the flag: pdpe1gb)
>
>
> 7- I am running most recent Xenial (4.4.0-13-generic), fully upgraded with
> "apt update ; apt full-upgrade ; reboot"
>
>
> 8- Huge pages on /proc/meminfo
>
> # grep Huge /proc/meminfo
> AnonHugePages: 26624 kB
> HugePages_Total: 2
> HugePages_Free: 1
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 1048576 kB
>
>
>
> 9- OVS process is running with DPDK options:
>
> ---
> ps xfa | grep dpdk
> 2634 ? S<s 0:00 ovs-vswit... --dpdk -c 0x1 -n 4 --socket-mem
> 1024 0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err
> -vfile:info --mlockall --no-chdir
> --log-file=/var/log/openvswitch/ovs-vswitchd.log
> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
> 2635 ? S<Ll 0:00 \_ ovs-vswitchd --dpdk -c 0x1 -n 4
> --socket-mem 1024 0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer
> -vsyslog:err -vfile:info --mlockall --no-chdir
> --log-file=/var/log/openvswitch/ovs-vswitchd.log
> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
> ---
>
>
> 10- DPDK-Compatible Driver activated
>
> ---
> dpdk_nic_bind --status
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:01:00.0 'NetXtreme II BCM57800 1/10 Gigabit Ethernet'
> drv=uio_pci_generic unused=bnx2x
> 0000:01:00.1 'NetXtreme II BCM57800 1/10 Gigabit Ethernet'
> drv=uio_pci_generic unused=bnx2x
> ---
>
>
> 11- OVS log have no "port 0" / "port 1"
>
> # service openvswitch-switch restart
> # journalctl | grep -i ovs:
>
> https://paste.ubuntu.com/15415651/
>
> Also, I'm not seeing any "PMD" lines on my OVS logs!
>
> NOTE: after running: "service openvswitch-switch restart", it still have
> dpdk options (step 9 above still valid)
>
>
> 12- The, OVS commands also fails...
>
> # ovs-vsctl show
> b70c1e0a-20d7-4bdb-98db-467330b72d07
> ovs_version: "2.5.0"
>
> # ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
> # ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> ovs-vsctl: Error detected while setting up 'dpdk0'. See ovs-vswitchd log
> for details.
>
> ---
>
> I just don't know what I am missing, I already checked the configurations
> dozens of times... At least, I'm not alone seeing this problems, since
> there are more users on Intel's forum with same issue...
>
> Thank you for your help! I'm going to IRC now.
>
> Best,
> Thiago
>
> On 18 March 2016 at 10:22, Christian Ehrhardt <
> christian.ehrhardt at canonical.com> wrote:
>
>> Hi Martin,
>> didn't find you on IRC today.
>> You might try to catch me later or on Monday if you like for faster
>> checks and comparison.
>>
>> I once more tried the ovs-dpdk setup.
>> While I'm blocked later on on other things I clearly can pass the stage
>> you are hanging on.
>>
>> I played a bit around and could cause the error you had by:
>> 1. not having any card on a dpdk compatible driver
>> 2. not enabling dpdk in ovs at all
>>
>> Could you please check your journalctl when starting openvswitch - it
>> should have a section like the following per card (ovs-)dpdk initializes:
>>
>> ovs-ctl[2155]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3
>> ovs-ctl[2155]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086
>> deviceID=0x1528
>> horsea ovs-ctl[2155]: EAL: PCI device 0000:04:00.1 on NUMA socket 0
>> horsea ovs-ctl[2155]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd
>> horsea ovs-ctl[2155]: EAL: PCI memory mapped at 0x7f5d40204000
>> horsea ovs-ctl[2155]: EAL: PCI memory mapped at 0x7f5d40404000
>> horsea ovs-vswitchd[2189]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3
>> horsea ovs-vswitchd[2189]: PMD: eth_ixgbe_dev_init(): port 0
>> vendorID=0x8086 deviceID=0x1528
>>
>> You see "port 0" that will be what is "dpdk0" later on.
>>
>> When I e.g. have two devices in ugb_pci_generic and start OVS I get one
>> for port 0 and one for port 1.
>> Then I can later on use dpdk0 and dpdk1, but not e.g. dpdk2
>>
>>
>>
>> Christian Ehrhardt
>> Software Engineer, Ubuntu Server
>> Canonical Ltd
>>
>> On Fri, Mar 11, 2016 at 10:17 PM, Martinx - ジェームズ <
>> thiagocmartinsc at gmail.com> wrote:
>>
>>> Hey Christian,
>>>
>>> Nice job on fixing many DPDK package bugs! :-D
>>>
>>> Replying inline below...
>>>
>>> On 9 March 2016 at 04:17, Christian Ehrhardt <
>>> christian.ehrhardt at canonical.com> wrote:
>>>
>>>> Thanks Jörg for making Martin and me aware.
>>>> It is kind of a close call, as much of that are new features, but you
>>>> are right - so I'll be moving this to the Ubuntu-sever mailing list.
>>>>
>>>> @Martin:
>>>> tl;dr - I suggest the following:
>>>> - you let me know if anything more comes back from the Intel forum post
>>>>
>>>
>>> Absolutely, I'll...
>>>
>>>
>>>> - short Term I'd ask you to reach out on IRC to me on #ubuntu-server -
>>>> my nick is "cpaelzer" (maybe we find something in an interactive session)
>>>>
>>>
>>> Sounds perfect! I'm there now, "tmartins" nick...
>>>
>>>
>>>> - somewhen in the next weeks I'll be back at openvswitch-dpdk. I'd
>>>> invite you to a IRC session or a hangout then (let me know what you would
>>>> prefer and I contact you once I'm ready)
>>>>
>>>
>>> That would be awesome!
>>>
>>>
>>>>
>>>>
>>>> Details:
>>>> The vfio-pci fix (and several others) are just waiting for the upload
>>>> to happen, if I don't run into any unexpected running blockers that should
>>>> be today.
>>>>
>>>
>>> Verified, it is working now.
>>>
>>>
>>>> Please let me know if you run into anything else with DPDK as I want to
>>>> clear as much as possible before release.
>>>>
>>>
>>> After latest release on Xenial, it looks very good! Both UIO and VFIO
>>> works without effort.
>>>
>>> I can easily switch between UIO and VFIO, that DPDK shows the NICs being
>>> used as "DPDK-Compatible drivers".
>>>
>>> Only OpenvSwitch+DPDK doesn't see any "dpdk0 | dpdk1" devices but, I
>>> know you guys are working into this...
>>>
>>>
>>>> Please feel free to just directly contact me about it - I'd suggest IRC
>>>> if that work for you.
>>>> We can move things to mailing lists whenever needed/applicable then.
>>>>
>>>
>>> Sounds great!
>>>
>>>
>>>>
>>>> Since last week I took a step back and work on more "low level" dpdk
>>>> testing and issues than the full setup with openvswitch-dpdk.
>>>> That helps the overall quality, but means my environment is currently
>>>> not useful to reproduce your case without too much reconfiguration.
>>>>
>>>
>>> That sounds like a good plan, I totally understand that... =)
>>>
>>>
>>>>
>>>> I highly appreciate to have someone in the Community working with me on
>>>> it.
>>>>
>>>
>>> You can count on me! I'm a big fan of Ubuntu...
>>>
>>> I met Mark in Vancouver on OpenStack summit, he is a great guy and it is
>>> easier now to see why Ubuntu is so cool! A lot of amazing people working
>>> here! ^_^
>>>
>>> So I wanted to ask if you could be patient for a few days, once my Env
>>>> is working for openvswitch-dpdk again I'd contact you and we could work
>>>> together comparing what break in your setup.
>>>>
>>>
>>> Of course! I'll be the beta tester. ;-)
>>>
>>> Christian Ehrhardt
>>>> Software Engineer, Ubuntu Server
>>>> Canonical Ltd
>>>>
>>>
>>> Just for the record, we moved this thread to Ubuntu Server, here is the
>>> previous thread on Ubuntu Devel, so, people on Ubuntu Server will
>>> understand what we're talking about:
>>>
>>>
>>> https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-March/016287.html
>>>
>>> Cheers!
>>> Thiago
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-server/attachments/20160319/a4f777b4/attachment.html>
More information about the ubuntu-server
mailing list