Xenial - OpenvSwitch with DPDK binding to 10G NIC, not working...
Christian Ehrhardt
christian.ehrhardt at canonical.com
Mon Mar 21 07:21:42 UTC 2016
Thanks Thiago,
I'll look into doability to enable that - even not thinking being after FF
there are pro's and con's to be considered.
We will go on with communication regarding this issue on that bug then.
Yet feel free to catch me on IRC as I suggested if you want.
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd
On Sat, Mar 19, 2016 at 8:01 AM, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
wrote:
> Hey Christian!
>
> I think I found the problem! I just filled a bug report:
>
> --
> Many PMD driver options are disabled, including "BNX2X"!
>
> https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1559408
> --
>
> Let me know about what do you think!
>
> Cheers!
> Thiago
>
> On 19 March 2016 at 02:49, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
> wrote:
>
>> Oh, I forgot to mention that my /etc/dpdk/dpdk.conf have:
>>
>> ---
>> NR_1G_PAGES=2
>> ---
>>
>> From what I'm seeing, my OpenvSwitch-DPDK is not using / seeing PMD.
>>
>> So, I'm researching about this but, I'll talk with you on Monday...
>>
>> Have a nice weekend Christian!
>>
>> Thanks,
>> Thiago
>>
>> On 18 March 2016 at 13:16, Martinx - ジェームズ <thiagocmartinsc at gmail.com>
>> wrote:
>>
>>> Hello Christian!
>>>
>>> Lets make a checklist...
>>>
>>>
>>> 1- Packages "dpdk and openvswitch-switch-dpdk", installed;
>>>
>>>
>>> 2- /etc/default/grub have:
>>>
>>> GRUB_CMDLINE_LINUX_DEFAULT="quiet splash default_hugepagesz=1GB
>>> hugepagesz=1G hugepages=4"
>>> GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 BOOTIF=01-b0-83-fe-d7-12-fc"
>>>
>>> Linux CMD Line:
>>>
>>> Command line: BOOT_IMAGE=/vmlinuz-4.4.0-13-generic
>>> root=/dev/mapper/xenial-1--vg-root ro console=ttyS0,115200n8
>>> BOOTIF=01-b0-83-fe-d7-12-fc quiet splash default_hugepagesz=1GB
>>> hugepagesz=1G hugepages=4 vt.handoff=7
>>>
>>>
>>> 3- /etc/dpdk/interfaces have:
>>>
>>> pci 0000:01:00.0 uio_pci_generic
>>> pci 0000:01:00.1 uio_pci_generic
>>>
>>>
>>> 4- /etc/default/openvswitch-switch have:
>>>
>>> DPDK_OPTS='--dpdk -c 0x1 -n 4 --socket-mem 1024,0'
>>>
>>>
>>> 5- /etc/alternatives/ovs-vswitchd: symbolic link to
>>> /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
>>>
>>>
>>> 6- My CPU supports 1G Hugepages (it have the flag: pdpe1gb)
>>>
>>>
>>> 7- I am running most recent Xenial (4.4.0-13-generic), fully upgraded
>>> with "apt update ; apt full-upgrade ; reboot"
>>>
>>>
>>> 8- Huge pages on /proc/meminfo
>>>
>>> # grep Huge /proc/meminfo
>>> AnonHugePages: 26624 kB
>>> HugePages_Total: 2
>>> HugePages_Free: 1
>>> HugePages_Rsvd: 0
>>> HugePages_Surp: 0
>>> Hugepagesize: 1048576 kB
>>>
>>>
>>>
>>> 9- OVS process is running with DPDK options:
>>>
>>> ---
>>> ps xfa | grep dpdk
>>> 2634 ? S<s 0:00 ovs-vswit... --dpdk -c 0x1 -n 4 --socket-mem
>>> 1024 0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err
>>> -vfile:info --mlockall --no-chdir
>>> --log-file=/var/log/openvswitch/ovs-vswitchd.log
>>> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
>>> 2635 ? S<Ll 0:00 \_ ovs-vswitchd --dpdk -c 0x1 -n 4
>>> --socket-mem 1024 0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer
>>> -vsyslog:err -vfile:info --mlockall --no-chdir
>>> --log-file=/var/log/openvswitch/ovs-vswitchd.log
>>> --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
>>> ---
>>>
>>>
>>> 10- DPDK-Compatible Driver activated
>>>
>>> ---
>>> dpdk_nic_bind --status
>>>
>>> Network devices using DPDK-compatible driver
>>> ============================================
>>> 0000:01:00.0 'NetXtreme II BCM57800 1/10 Gigabit Ethernet'
>>> drv=uio_pci_generic unused=bnx2x
>>> 0000:01:00.1 'NetXtreme II BCM57800 1/10 Gigabit Ethernet'
>>> drv=uio_pci_generic unused=bnx2x
>>> ---
>>>
>>>
>>> 11- OVS log have no "port 0" / "port 1"
>>>
>>> # service openvswitch-switch restart
>>> # journalctl | grep -i ovs:
>>>
>>> https://paste.ubuntu.com/15415651/
>>>
>>> Also, I'm not seeing any "PMD" lines on my OVS logs!
>>>
>>> NOTE: after running: "service openvswitch-switch restart", it still have
>>> dpdk options (step 9 above still valid)
>>>
>>>
>>> 12- The, OVS commands also fails...
>>>
>>> # ovs-vsctl show
>>> b70c1e0a-20d7-4bdb-98db-467330b72d07
>>> ovs_version: "2.5.0"
>>>
>>> # ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
>>> # ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
>>> ovs-vsctl: Error detected while setting up 'dpdk0'. See ovs-vswitchd
>>> log for details.
>>>
>>> ---
>>>
>>> I just don't know what I am missing, I already checked the
>>> configurations dozens of times... At least, I'm not alone seeing this
>>> problems, since there are more users on Intel's forum with same issue...
>>>
>>> Thank you for your help! I'm going to IRC now.
>>>
>>> Best,
>>> Thiago
>>>
>>> On 18 March 2016 at 10:22, Christian Ehrhardt <
>>> christian.ehrhardt at canonical.com> wrote:
>>>
>>>> Hi Martin,
>>>> didn't find you on IRC today.
>>>> You might try to catch me later or on Monday if you like for faster
>>>> checks and comparison.
>>>>
>>>> I once more tried the ovs-dpdk setup.
>>>> While I'm blocked later on on other things I clearly can pass the stage
>>>> you are hanging on.
>>>>
>>>> I played a bit around and could cause the error you had by:
>>>> 1. not having any card on a dpdk compatible driver
>>>> 2. not enabling dpdk in ovs at all
>>>>
>>>> Could you please check your journalctl when starting openvswitch - it
>>>> should have a section like the following per card (ovs-)dpdk initializes:
>>>>
>>>> ovs-ctl[2155]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3
>>>> ovs-ctl[2155]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086
>>>> deviceID=0x1528
>>>> horsea ovs-ctl[2155]: EAL: PCI device 0000:04:00.1 on NUMA socket 0
>>>> horsea ovs-ctl[2155]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd
>>>> horsea ovs-ctl[2155]: EAL: PCI memory mapped at 0x7f5d40204000
>>>> horsea ovs-ctl[2155]: EAL: PCI memory mapped at 0x7f5d40404000
>>>> horsea ovs-vswitchd[2189]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3
>>>> horsea ovs-vswitchd[2189]: PMD: eth_ixgbe_dev_init(): port 0
>>>> vendorID=0x8086 deviceID=0x1528
>>>>
>>>> You see "port 0" that will be what is "dpdk0" later on.
>>>>
>>>> When I e.g. have two devices in ugb_pci_generic and start OVS I get one
>>>> for port 0 and one for port 1.
>>>> Then I can later on use dpdk0 and dpdk1, but not e.g. dpdk2
>>>>
>>>>
>>>>
>>>> Christian Ehrhardt
>>>> Software Engineer, Ubuntu Server
>>>> Canonical Ltd
>>>>
>>>> On Fri, Mar 11, 2016 at 10:17 PM, Martinx - ジェームズ <
>>>> thiagocmartinsc at gmail.com> wrote:
>>>>
>>>>> Hey Christian,
>>>>>
>>>>> Nice job on fixing many DPDK package bugs! :-D
>>>>>
>>>>> Replying inline below...
>>>>>
>>>>> On 9 March 2016 at 04:17, Christian Ehrhardt <
>>>>> christian.ehrhardt at canonical.com> wrote:
>>>>>
>>>>>> Thanks Jörg for making Martin and me aware.
>>>>>> It is kind of a close call, as much of that are new features, but you
>>>>>> are right - so I'll be moving this to the Ubuntu-sever mailing list.
>>>>>>
>>>>>> @Martin:
>>>>>> tl;dr - I suggest the following:
>>>>>> - you let me know if anything more comes back from the Intel forum
>>>>>> post
>>>>>>
>>>>>
>>>>> Absolutely, I'll...
>>>>>
>>>>>
>>>>>> - short Term I'd ask you to reach out on IRC to me on #ubuntu-server
>>>>>> - my nick is "cpaelzer" (maybe we find something in an interactive session)
>>>>>>
>>>>>
>>>>> Sounds perfect! I'm there now, "tmartins" nick...
>>>>>
>>>>>
>>>>>> - somewhen in the next weeks I'll be back at openvswitch-dpdk. I'd
>>>>>> invite you to a IRC session or a hangout then (let me know what you would
>>>>>> prefer and I contact you once I'm ready)
>>>>>>
>>>>>
>>>>> That would be awesome!
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> Details:
>>>>>> The vfio-pci fix (and several others) are just waiting for the upload
>>>>>> to happen, if I don't run into any unexpected running blockers that should
>>>>>> be today.
>>>>>>
>>>>>
>>>>> Verified, it is working now.
>>>>>
>>>>>
>>>>>> Please let me know if you run into anything else with DPDK as I want
>>>>>> to clear as much as possible before release.
>>>>>>
>>>>>
>>>>> After latest release on Xenial, it looks very good! Both UIO and VFIO
>>>>> works without effort.
>>>>>
>>>>> I can easily switch between UIO and VFIO, that DPDK shows the NICs
>>>>> being used as "DPDK-Compatible drivers".
>>>>>
>>>>> Only OpenvSwitch+DPDK doesn't see any "dpdk0 | dpdk1" devices but, I
>>>>> know you guys are working into this...
>>>>>
>>>>>
>>>>>> Please feel free to just directly contact me about it - I'd suggest
>>>>>> IRC if that work for you.
>>>>>> We can move things to mailing lists whenever needed/applicable then.
>>>>>>
>>>>>
>>>>> Sounds great!
>>>>>
>>>>>
>>>>>>
>>>>>> Since last week I took a step back and work on more "low level" dpdk
>>>>>> testing and issues than the full setup with openvswitch-dpdk.
>>>>>> That helps the overall quality, but means my environment is currently
>>>>>> not useful to reproduce your case without too much reconfiguration.
>>>>>>
>>>>>
>>>>> That sounds like a good plan, I totally understand that... =)
>>>>>
>>>>>
>>>>>>
>>>>>> I highly appreciate to have someone in the Community working with me
>>>>>> on it.
>>>>>>
>>>>>
>>>>> You can count on me! I'm a big fan of Ubuntu...
>>>>>
>>>>> I met Mark in Vancouver on OpenStack summit, he is a great guy and it
>>>>> is easier now to see why Ubuntu is so cool! A lot of amazing people working
>>>>> here! ^_^
>>>>>
>>>>> So I wanted to ask if you could be patient for a few days, once my Env
>>>>>> is working for openvswitch-dpdk again I'd contact you and we could work
>>>>>> together comparing what break in your setup.
>>>>>>
>>>>>
>>>>> Of course! I'll be the beta tester. ;-)
>>>>>
>>>>> Christian Ehrhardt
>>>>>> Software Engineer, Ubuntu Server
>>>>>> Canonical Ltd
>>>>>>
>>>>>
>>>>> Just for the record, we moved this thread to Ubuntu Server, here is
>>>>> the previous thread on Ubuntu Devel, so, people on Ubuntu Server will
>>>>> understand what we're talking about:
>>>>>
>>>>>
>>>>> https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-March/016287.html
>>>>>
>>>>> Cheers!
>>>>> Thiago
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-server/attachments/20160321/332e39e4/attachment.html>
More information about the ubuntu-server
mailing list