<div dir="ltr">Hello Christian!<div><br></div><div>Lets make a checklist...</div><div><br></div><div><br></div><div>1- Packages "dpdk and openvswitch-switch-dpdk", installed;</div><div><br></div><div><br></div><div>2- /etc/default/grub have:</div><div><br></div><div>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash default_hugepagesz=1GB hugepagesz=1G hugepages=4"<br></div><div>GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 BOOTIF=01-b0-83-fe-d7-12-fc"<br></div><div><br></div><div>Linux CMD Line:</div><div><br></div><div>Command line: BOOT_IMAGE=/vmlinuz-4.4.0-13-generic root=/dev/mapper/xenial-1--vg-root ro console=ttyS0,115200n8 BOOTIF=01-b0-83-fe-d7-12-fc quiet splash default_hugepagesz=1GB hugepagesz=1G hugepages=4 vt.handoff=7<br></div><div><br></div><div><br></div><div>3- /etc/dpdk/interfaces have:</div><div><br></div><div><div>pci 0000:01:00.0 uio_pci_generic</div><div>pci 0000:01:00.1 uio_pci_generic</div><div><br></div></div><div><br></div><div>4- /etc/default/openvswitch-switch have:</div><div><br></div><div>DPDK_OPTS='--dpdk -c 0x1 -n 4 --socket-mem 1024,0'<br></div><div><br></div><div><br></div><div>5- /etc/alternatives/ovs-vswitchd: symbolic link to /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk</div><div><br></div><div><br></div><div>6- My CPU supports 1G Hugepages (it have the flag: pdpe1gb)</div><div><br></div><div><br></div><div>7- I am running most recent Xenial (4.4.0-13-generic), fully upgraded with "apt update ; apt full-upgrade ; reboot"</div><div><br></div><div><br></div><div>8- Huge pages on /proc/meminfo</div><div><br></div><div><div># grep Huge /proc/meminfo</div><div>AnonHugePages: 26624 kB</div><div>HugePages_Total: 2</div><div>HugePages_Free: 1</div><div>HugePages_Rsvd: 0</div><div>HugePages_Surp: 0</div><div>Hugepagesize: 1048576 kB</div></div><div><br></div><div><br></div><div><br></div><div>9- OVS process is running with DPDK options:</div><div><br></div><div>---</div><div><div>ps xfa | grep dpdk</div><div> 2634 ? S<s 0:00 ovs-vswit... --dpdk -c 0x1 -n 4 --socket-mem 1024 0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor</div><div> 2635 ? S<Ll 0:00 \_ ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 1024 0 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor</div></div><div>---</div><div><br></div><div><br></div><div>10- DPDK-Compatible Driver activated</div><div><br></div><div>---</div><div><div>dpdk_nic_bind --status</div><div><br></div><div>Network devices using DPDK-compatible driver</div><div>============================================</div><div>0000:01:00.0 'NetXtreme II BCM57800 1/10 Gigabit Ethernet' drv=uio_pci_generic unused=bnx2x</div><div>0000:01:00.1 'NetXtreme II BCM57800 1/10 Gigabit Ethernet' drv=uio_pci_generic unused=bnx2x</div></div><div>---</div><div><br></div><div><br></div><div>11- OVS log have no "port 0" / "port 1"</div><div><br></div><div># service openvswitch-switch restart</div><div># journalctl | grep -i ovs:</div><div><br></div><div><a href="https://paste.ubuntu.com/15415651/">https://paste.ubuntu.com/15415651/</a><br></div><div><br></div><div>Also, I'm not seeing any "PMD" lines on my OVS logs!</div><div><br></div><div>NOTE: after running: "service openvswitch-switch restart", it still have dpdk options (step 9 above still valid)</div><div><br></div><div><br></div><div>12- The, OVS commands also fails...</div><div><br></div><div><div># ovs-vsctl show</div><div>b70c1e0a-20d7-4bdb-98db-467330b72d07</div><div> ovs_version: "2.5.0"</div></div><div><br></div><div># ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev</div><div># ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk</div><div>ovs-vsctl: Error detected while setting up 'dpdk0'. See ovs-vswitchd log for details.</div><div><br></div><div>---</div><div><br></div><div>I just don't know what I am missing, I already checked the configurations dozens of times... At least, I'm not alone seeing this problems, since there are more users on Intel's forum with same issue...</div><div><br></div><div>Thank you for your help! I'm going to IRC now.</div><div><br></div><div>Best,</div><div>Thiago</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 18 March 2016 at 10:22, Christian Ehrhardt <span dir="ltr"><<a href="mailto:christian.ehrhardt@canonical.com" target="_blank">christian.ehrhardt@canonical.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Martin,<div>didn't find you on IRC today.</div><div>You might try to catch me later or on Monday if you like for faster checks and comparison.</div><div><br></div><div>I once more tried the ovs-dpdk setup.</div><div>While I'm blocked later on on other things I clearly can pass the stage you are hanging on.</div><div><br></div><div>I played a bit around and could cause the error you had by:<br>1. not having any card on a dpdk compatible driver</div><div>2. not enabling dpdk in ovs at all</div><div><br></div><div>Could you please check your journalctl when starting openvswitch - it should have a section like the following per card (ovs-)dpdk initializes:</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">ovs-ctl[2155]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3 </span><br>ovs-ctl[2155]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x1528 <br>horsea ovs-ctl[2155]: EAL: PCI device 0000:04:00.1 on NUMA socket 0 <br>horsea ovs-ctl[2155]: EAL: probe driver: 8086:1528 rte_ixgbe_pmd <br>horsea ovs-ctl[2155]: EAL: PCI memory mapped at 0x7f5d40204000 <br>horsea ovs-ctl[2155]: EAL: PCI memory mapped at 0x7f5d40404000 <br>horsea ovs-vswitchd[2189]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3 <br>horsea ovs-vswitchd[2189]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x1528</span></div><div><br></div><div>You see "port 0" that will be what is "dpdk0" later on.</div><div><br></div><div>When I e.g. have two devices in ugb_pci_generic and start OVS I get one for port 0 and one for port 1.</div><div>Then I can later on use dpdk0 and dpdk1, but not e.g. dpdk2</div><div><br></div><div><br></div></div><div class="gmail_extra"><span class=""><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(136,136,136);font-size:12.8px">Christian Ehrhardt</span><div style="color:rgb(136,136,136);font-size:12.8px">Software Engineer, Ubuntu Server</div><div style="color:rgb(136,136,136);font-size:12.8px">Canonical Ltd</div></div></div></div></div></div> <br></span><div><div class="h5"><div class="gmail_quote">On Fri, Mar 11, 2016 at 10:17 PM, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Christian,<div><br></div><div> Nice job on fixing many DPDK package bugs! :-D</div><div><br></div><div> Replying inline below...</div><div class="gmail_extra"><br><div class="gmail_quote"><span>On 9 March 2016 at 04:17, Christian Ehrhardt <span dir="ltr"><<a href="mailto:christian.ehrhardt@canonical.com" target="_blank">christian.ehrhardt@canonical.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Thanks Jörg for making Martin and me aware.<div>It is kind of a close call, as much of that are new features, but you are right - so I'll be moving this to the Ubuntu-sever mailing list.</div><div><br></div><div>@Martin:</div><div><div>tl;dr - I suggest the following:</div><div>- you let me know if anything more comes back from the Intel forum post</div></div></div></blockquote><div><br></div></span><div>Absolutely, I'll...</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div>- short Term I'd ask you to reach out on IRC to me on #ubuntu-server - my nick is "cpaelzer" (maybe we find something in an interactive session)<br></div></div></div></blockquote><div><br></div></span><div>Sounds perfect! I'm there now, "tmartins" nick...</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div></div><div>- somewhen in the next weeks I'll be back at openvswitch-dpdk. I'd invite you to a IRC session or a hangout then (let me know what you would prefer and I contact you once I'm ready)</div></div></div></blockquote><div><br></div></span><div>That would be awesome!</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div><div>Details:</div><div>The vfio-pci fix (and several others) are just waiting for the upload to happen, if I don't run into any unexpected running blockers that should be today.</div></div></blockquote><div><br></div></span><div>Verified, it is working now.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Please let me know if you run into anything else with DPDK as I want to clear as much as possible before release.</div></div></blockquote><div><br></div></span><div>After latest release on Xenial, it looks very good! Both UIO and VFIO works without effort.</div><div><br></div><div>I can easily switch between UIO and VFIO, that DPDK shows the NICs being used as "DPDK-Compatible drivers".</div><div><br></div><div>Only OpenvSwitch+DPDK doesn't see any "dpdk0 | dpdk1" devices but, I know you guys are working into this...</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Please feel free to just directly contact me about it - I'd suggest IRC if that work for you.</div><div>We can move things to mailing lists whenever needed/applicable then.</div></div></blockquote><div><br></div></span><div>Sounds great!</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Since last week I took a step back and work on more "low level" dpdk testing and issues than the full setup with openvswitch-dpdk.</div><div>That helps the overall quality, but means my environment is currently not useful to reproduce your case without too much reconfiguration.</div></div></blockquote><div><br></div></span><div>That sounds like a good plan, I totally understand that... =)</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>I highly appreciate to have someone in the Community working with me on it.</div></div></blockquote><div><br></div></span><div>You can count on me! I'm a big fan of Ubuntu...</div><div><br></div><div>I met Mark in Vancouver on OpenStack summit, he is a great guy and it is easier now to see why Ubuntu is so cool! A lot of amazing people working here! ^_^</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>So I wanted to ask if you could be patient for a few days, once my Env is working for openvswitch-dpdk again I'd contact you and we could work together comparing what break in your setup.</div></div></blockquote><div><br></div></span><div>Of course! I'll be the beta tester. ;-)</div><span><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="gmail_extra"><span><div><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(136,136,136);font-size:12.8px">Christian Ehrhardt</span><div style="color:rgb(136,136,136);font-size:12.8px">Software Engineer, Ubuntu Server</div><div style="color:rgb(136,136,136);font-size:12.8px">Canonical Ltd</div></div></div></div></div></div></span></div></blockquote><div><br></div></span><div>Just for the record, we moved this thread to Ubuntu Server, here is the previous thread on Ubuntu Devel, so, people on Ubuntu Server will understand what we're talking about:</div><div><br></div><div><a href="https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-March/016287.html" target="_blank">https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-March/016287.html</a><br></div><div><br></div><div>Cheers!</div><div>Thiago</div></div></div></div> </blockquote></div><br></div></div></div> </blockquote></div><br></div>