<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 8 March 2016 at 20:59, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hey Christian,<div><br></div><div>Thank you for your hard work!</div><div><br></div><div>Replying inline below...</div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On 8 March 2016 at 04:48, Christian Ehrhardt <span dir="ltr"><<a href="mailto:christian.ehrhardt@canonical.com" target="_blank">christian.ehrhardt@canonical.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Hi Thiago,</div><div>comments inline below ...</div><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(136,136,136);font-size:12.8px">Christian Ehrhardt</span><div style="color:rgb(136,136,136);font-size:12.8px">Software Engineer, Ubuntu Server</div><div style="color:rgb(136,136,136);font-size:12.8px">Canonical Ltd</div></div></div></div></div></div>
<div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><br>From: <b class="gmail_sendername">Martinx - ジェームズ</b> <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span><br>Date: Mon, Mar 7, 2016 at 2:28 PM<br>Subject: Re: Xenial - OpenvSwitch with DPDK binding to 10G NIC, not working...<br>To: Ubuntu Core developers <<a href="mailto:ubuntu-devel-discuss@lists.ubuntu.com" target="_blank">ubuntu-devel-discuss@lists.ubuntu.com</a>><br><br><br><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On 7 March 2016 at 16:40, Martinx - ジェームズ <span dir="ltr"><<a href="mailto:thiagocmartinsc@gmail.com" target="_blank">thiagocmartinsc@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hey guys,<div><br></div><div> I'm trying to create two OVS bridges powered by DPDK, on Xenial, but I'm facing problems...</div><div><br></div><div> First, there is no documentation about how to do this... So far, I found the following:</div><div><br></div><div><br></div><div><div> - This link doesn't how to bind real NIC:</div></div><div> <a href="https://software.intel.com/en-us/articles/using-open-vswitch-with-dpdk-on-ubuntu" target="_blank">https://software.intel.com/en-us/articles/using-open-vswitch-with-dpdk-on-ubuntu</a><br></div><div><br></div><div> - This one isn't specific to Ubuntu OVS + DPDK packages:</div><div> <a href="https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md" target="_blank">https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md</a></div><div><br></div><div> - Similar with above but, very old... Also not related to Ubuntu packages:</div><div> <a href="https://feiskyer.github.io/2015/03/01/open-vswitch-over-dpdk-on-ubuntu/" target="_blank">https://feiskyer.github.io/2015/03/01/open-vswitch-over-dpdk-on-ubuntu/</a></div><div><br></div></div></blockquote><div><br></div></span><div>Yeah docs are often outdated, I have realized that as well.</div><div>Not because people are lazy, but because the environment is changing so fast.</div><div>It is already planned to do a blog post or wiki or such along 16.04 release to guide users of the very updated dpdk / openvsiwtch-dpdk.</div><div>But for now we are working on so many issues that things are changing too fast to pre-write that document.</div><div><br></div><div>To add a bit I also found this one useful:</div><div><a href="http://www.ran-lifshitz.com/2015/08/28/open-vswitch-netdev-dpdk-with-vhost-user-support-update/" target="_blank">http://www.ran-lifshitz.com/2015/08/28/open-vswitch-netdev-dpdk-with-vhost-user-support-update/</a><br></div><div>But be aware of</div><div><a href="https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1546565" target="_blank">https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1546565</a><br></div><div>if you want to pass it to guests.</div><div>At least there the log entries will be pretty obvious with "permission denied" on the vhost-user sockets.</div></div></div></div></div></div></div></div></blockquote><div><br></div></div></div><div>Yes, I totally understand that. I want to be ready for 16.04 release, since day one. Currently, my company works with CentOS and manually installed DPDK, which is just plain terrible, and I am pushing Ubuntu!</div><div><br></div><div>I'm not facing bug #1546565, I'm not there yet.</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div><div> So, to begin with, I start with Intel.com guide, it worked! But without real NIC.</div><div><br></div><div> Then, I tried to bind it to real NIC, like this:</div><div><br></div><div> Added two NIC PCI id to "/etc/dpdk/interfaces" file:</div><div><br></div><div>---</div><div><div>pci   0000:01:00.0   uio-pci-generic</div><div>pci   0000:01:00.1   uio-pci-generic</div></div><div>---</div><div><br></div><div> Then, reboot, I can see both NICs using "DPDK-compatible driver" by running: "dpdk_nic_bind --status".</div><div><br></div><div> Trying to create new OVS Bridges with DPDK devices, failing:</div><div><br></div><div>---</div><div><div>root@xenial-1:~# ovs-vsctl add-br ovsbr0</div><div>root@xenial-1:~# ovs-vsctl set bridge ovsbr0 datapath_type=netdev</div><div>root@xenial-1:~# ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk</div><div>ovs-vsctl: Error detected while setting up 'dpdk0'. See ovs-vswitchd log for details.</div></div><div></div></div></blockquote></span><div>[...] </div><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>2016-03-07T19:34:07.532Z|00136|bridge|WARN|could not open network device dpdk0 (No such device)<br></div><div></div></div></blockquote></span><div>[...] </div><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>    Port "dpdk0"<br></div><div><div>      Interface "dpdk0"</div><div>        type: dpdk</div><div>        error: "could not open network device dpdk0 (No such device)"</div></div></div></blockquote><div> </div></span><div>I had seen something similar.</div><div>The particular one you are describing here was for me a device that was either no bound by dpdk or had no compatible driver.</div><div>But according to what I see both should be ok for you.</div></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>Yes, I have DPDK compatible 10G NIC cards, I can bind to both uio-generic and vfio-pci but, OpenvSwitch+DPDK doesn't find it.</div><div><br></div><div>The following command always fails:</div><span class=""><div><br></div><div>ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk<br></div><div><br></div></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><span class=""><div>Often I realized that the vswitch log alone isn't too helpful.</div><div>I recommend using journalctl along to get the EAL messages of dpdk as well.</div></span></div></div></div></div></div></div></div></blockquote><div><br></div><div>Sure, I agree.</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>To help you I created a merged log (time goes top down) which has both logs and the command I used for a successful ovs-dpdk port add.</div><div>I think that will help you to see where your systems behaviour starts to differ.</div><div><br></div><div>=> <a href="http://paste.ubuntu.com/15326358/" target="_blank">http://paste.ubuntu.com/15326358/</a></div><div><br></div><div>In your case you might e.g. see "skipped device" there leading eventually to your dpdk0 not found issue.</div><div><br></div><div>Also you might already have done that, but just not mentioned. Still to help as much as possible - have you set DPDK_OPTS, I for example did that before (re)starting it?</div><div>For example</div><div>   echo "set DPDK_OPTS='--dpdk -c 0x1 -n 4 -m 4096'" >> /etc/default/openvswitch-switch</div></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>It is good to see that it is working for you! Maybe I am almost there...</div><div><br></div><div>Yes, I tried all of this (specially <a href="https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md" target="_blank">INSTALL.DPDK.md</a>)...</div><div><br></div><div>I can see that [ps xufawww] OVS is running with --dpdk options.</div><div><br></div><div>I'll try now, to follow your pastebin step by step...</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div><br></div><div>[...]</div><div><br></div><div>I have also seen the part about vfio-pci in your mail and thank you for reporting the bug, I'll handle work and communication of this issue in the bug.</div></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>Awesome! I'm see that you already fixed it! That was fast!  :-D</div><div><br></div><div>Thank you again!</div><div><br></div><div>Cheers!</div><div>Thiago </div></div></div></div></blockquote><div><br></div><div>Yeah, still doesn't work... I am unable to reproduce what you did here: <a href="http://paste.ubuntu.com/15326358/" target="_blank">http://paste.ubuntu.com/15326358/</a></div><div><br></div><div>I'm seeing the problem over and over, no matter which driver I'm using uio-pci-generic or vfio-pci...</div><div><br></div><div>I'll wait for instructions, no problem... Neither Intel guys knows how to do this on Ubuntu, so, who am I?  :-P</div><div><br></div><div>My post on Intel forum is now available (moderated):</div><div><br></div><div><a href="https://software.intel.com/en-us/forums/networking/topic/611450">https://software.intel.com/en-us/forums/networking/topic/611450</a><br></div><div><br></div><div>Cheers man!</div><div><br></div><div>Thanks,</div><div>Thiago</div></div><br></div></div>