[Bug 1680956] Re: Fail to launch an OpenStack Instance with hugepages on top of OVS+DPDK

[ChristianEhrhardt]

I checked the DPDK code which creates the extra mountpoint.
It only creates the extra mountpoints if
1. a given size e.g 1G is configured in dpdk.conf
2. no mountpoint for this size exists

In general a system always has /dev/hugepages at the "default" hugepagesz.
The default size can be changed as kernel parm, so this might be either one or another size.

Since DPDK wants to provide a simple way to get users going with it the hugepage setup is somewhat automated and working for non complex cases as outlined by the comment in the config file.
So if one is e.g. setting NR_1G_PAGES and 1G is not the current default size it will create a mountpoint. There the format is /dev/hugepages-${sizeinkb} then.

Qemu-kvm at the same time creates /run/hugepages/kvm if KVM_HUGEPAGES is
set.

To some extend that means if somebody adds more hugepage mountpoints to the system either via DPDK or otherwise this is a config change that needs other config changes to work well.
IMHO in this case one should set the hugetlbfs_mount in /etc/libvirt/qemu.conf - 
especially since this path is already allowed for apparmor.

Since we have a statistical reason to cause this I'll add it to the
libvirt profile, but in general that is more a configuration issue than
a bug.

In addition to my questions in comment #6 I'd ask to get a copy of
/etc/dpdk/dpdk.conf in your case if possible.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1680956

Title:
  Fail to launch an OpenStack Instance with hugepages on top of OVS+DPDK

Status in Ubuntu Cloud Archive:
  New
Status in libvirt package in Ubuntu:
  Incomplete

Bug description:
  Guys,

   I'm trying to launch an Instance on OpenStack Ocata, using Ubuntu
  16.04, on top of OpenvSwitch with DPDK, and OVN.

   I can launch an Instance without hugepages but, when I change
  OpenStack's Flavor for it to have: "hw:mem_size_pages=large", the
  following error appear on nova-compute.log:

  
  ---
  2017-04-07 20:13:18.504 3996 ERROR nova.compute.manager [instance: fbbb7bde-763e-40d3-8987-fb3a6c568ad8] libvirtError: internal error: process exited while connecting to monitor: 2017-04-07T20:13:17.764939Z qemu-system-x86_64: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages-1048576/libvirt/qemu,share=yes,size=4294967296,host-nodes=0,policy=bind: can't open backing store /dev/hugepages-1048576/libvirt/qemu for guest RAM: Permission denied
  ---

  At syslog:

  ---
  Apr 7 20:13:17 expert-jennet kernel: [ 1535.437956] audit: type=1400 audit(1491595997.759:23): apparmor="DENIED" operation="mknod" profile="libvirt-fbbb7bde-763e-40d3-8987-fb3a6c568ad8" name="/dev/hugepages-1048576/libvirt/qemu/qemu_back_mem._objects_ram-node0.BFNx2W" pid=8341 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=64055 ouid=64055
  ---

  Maybe it is related to:

  https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1524737

  But I'm not sure if it is relate to 1524737 or not...

  Let me know if there are any workarounds available!

  Thanks,
  Thiago

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: libvirt0 2.5.0-3ubuntu5~cloud0 [origin: Canonical]
  ProcVersionSignature: Ubuntu 4.8.0-46.49~16.04.1-generic 4.8.17
  Uname: Linux 4.8.0-46-generic x86_64
  ApportVersion: 2.20.1-0ubuntu2.5
  Architecture: amd64
  CrashDB:
   {
                  "impl": "launchpad",
                  "project": "cloud-archive",
                  "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
               }
  Date: Fri Apr  7 20:27:27 2017
  ProcEnviron:
   TERM=screen
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: libvirt
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1680956/+subscriptions