[Bug 1680956] Re: Fail to launch an OpenStack Instance with hugepages on top of OVS+DPDK
ChristianEhrhardt
1680956 at bugs.launchpad.net
Tue Apr 18 12:52:58 UTC 2017
As I discussed with Nish (thanks for the update here) we might need to extend the apparmor profile - but we should understand the case better before doing so.
I use KVM guests on systems with DPDK hugepages as well without issues.
And I use huge pages for them so something has to be special in the charms configuring them here.
If you enable hugepages for DPDK it will make sure there are hugepage
mounts for each kind of page sizes. So far we haven't had an issue with
it but it seems if on the same system you run qemu with huge pages this
issue occurs.
Usually I set a 1 in /etc/default/qemu-kvm:
9 # Set this to 1 if you want hugepages to be available to kvm under
10 # /run/hugepages/kvm
11 KVM_HUGEPAGES=0
And my guests have:
<memoryBacking>
<hugepages>
<page size="2" unit="M" nodeset="0"/>
</hugepages>
</memoryBacking>
This setup is working fine so far, but especially the explicit page size
is a consequence of the support for multiple page sizes by libvirt/qemu
and might be missing in the charm/openstack so far.
If the option KVM_HUGEPAGES is set qemu-kvm ensures there is a mountpoint (of the default huge page size) in /run/hugepages/kvm
mkdir -p /run/hugepages/kvm
mount -t hugetlbfs hugetlbfs-kvm -o mode=775,gid=kvm /run/hugepages/kvm
BTW "owner "/run/hugepages/kvm/libvirt/qemu/**" rw," is also in the
apparmor profile.
But libvirt without explicit config will pick on hugepages from /proc/mounts.
But in some sense that means would need to include any target dir hugepages will ever be mounted to. Of course we will might add the wildcard for the DPDK paths, but still that leaves the issue open for any other path added by e.g. an Database admin for Hugepages to his DB.
In fact there is a config that can set this as needed in /etc/libvirt/qemu.conf
Quoting from that config:
If provided by the host and a hugetlbfs mount point is configured,
a guest may request huge page backing. When this mount point is
unspecified here, determination of a host mount point in /proc/mounts
will be attempted. Specifying an explicit mount overrides detection
of the same in /proc/mounts. Setting the mount point to "" will
disable guest hugepage backing. If desired, multiple mount points can
be specified at once, separated by comma and enclosed in square
brackets, for example:
[...]
It might be worth to set something in there as a workaround to disable the suboptimal detection.
$ echo 'hugetlbfs_mount = ["/run/hugepages/kvm"]' >> /etc/libvirt/qemu.conf
Essentially qemu only "does what it is told" like in
-object memory-backend-file,id=mem1,size=1G,mem-path=/mnt/hugepages-1G \
-device pc-dimm,id=dimm1,memdev=mem1 \
So it might be interesting to see what commandline libvirt creates.
Questions:
- Was there any chance to test the workaround Seth suggested?
- Thiago, could you attach a copy of the target systems /etc/default/qemu-kvm and the created guest XML that openstack is pushing?
- Also in your host could you do "mount | grep huge" and report that as well, maybe the order is important.
- can you report the qemu commandline that is constructed by libvirt from /var/log/libvier/qemu/<guestname>.log?
- Could you try the workaround to disable libvirts detection by setting hugetlbfs_mount in /etc/libvirt/qemu.conf
Based on that feedback we can decide if we "just" want to widen the
apparmor profile, or to modify the charms or if we need more than that.
** Changed in: libvirt (Ubuntu)
Status: New => Incomplete
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1680956
Title:
Fail to launch an OpenStack Instance with hugepages on top of OVS+DPDK
Status in Ubuntu Cloud Archive:
New
Status in libvirt package in Ubuntu:
Incomplete
Bug description:
Guys,
I'm trying to launch an Instance on OpenStack Ocata, using Ubuntu
16.04, on top of OpenvSwitch with DPDK, and OVN.
I can launch an Instance without hugepages but, when I change
OpenStack's Flavor for it to have: "hw:mem_size_pages=large", the
following error appear on nova-compute.log:
---
2017-04-07 20:13:18.504 3996 ERROR nova.compute.manager [instance: fbbb7bde-763e-40d3-8987-fb3a6c568ad8] libvirtError: internal error: process exited while connecting to monitor: 2017-04-07T20:13:17.764939Z qemu-system-x86_64: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages-1048576/libvirt/qemu,share=yes,size=4294967296,host-nodes=0,policy=bind: can't open backing store /dev/hugepages-1048576/libvirt/qemu for guest RAM: Permission denied
---
At syslog:
---
Apr 7 20:13:17 expert-jennet kernel: [ 1535.437956] audit: type=1400 audit(1491595997.759:23): apparmor="DENIED" operation="mknod" profile="libvirt-fbbb7bde-763e-40d3-8987-fb3a6c568ad8" name="/dev/hugepages-1048576/libvirt/qemu/qemu_back_mem._objects_ram-node0.BFNx2W" pid=8341 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=64055 ouid=64055
---
Maybe it is related to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1524737
But I'm not sure if it is relate to 1524737 or not...
Let me know if there are any workarounds available!
Thanks,
Thiago
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: libvirt0 2.5.0-3ubuntu5~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.8.0-46.49~16.04.1-generic 4.8.17
Uname: Linux 4.8.0-46-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
CrashDB:
{
"impl": "launchpad",
"project": "cloud-archive",
"bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
}
Date: Fri Apr 7 20:27:27 2017
ProcEnviron:
TERM=screen
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1680956/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list