[Bug 614322] Re: libvirt not recognizing NUMA architecture

[EAB]

I can confirm this.

Most of our hosts have 64GB RAM and 2 Intel Nehalem hexa-cores.
With 3 VM's using 16GB RAM each.
I noticed performance degredation and the host also swapped out a lot of memory. The swapping degraded performance dramatically.
vm.swappiness=0 in /etc/sysctl.conf did not help.

It seems that NUMA on Intel CPU's can be expensive because RAM needs to
be transfered from other nodes. With only 1 node (socket) there is no
problem. With 2 or more nodes you see slower.

Without the capabilities you can prevent this behavior by pinning the vcpu's.
You should spread you VM's over the available nodes:

numactl --hardware | grep "node 0 cpus"
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22

Your XML should contain something like this:
<vcpu cpuset='0,2,4,6,8,10,12,14,16,18,20,22'>1</vcpu> 
<vcpu cpuset='0,2,4,6,8,10,12,14,16,18,20,22'>2</vcpu> 
<vcpu cpuset='0,2,4,6,8,10,12,14,16,18,20,22'>3</vcpu> 
<vcpu cpuset='0,2,4,6,8,10,12,14,16,18,20,22'>4</vcpu> 

The next VM should use they other node.
numactl --hardware | grep "node 1 cpus"
node 0 cpus: 1 3 5 7 9 11 13 15 17 19 21 23
<vcpu cpuset='1,3,5,7,9,11,13,15,17,19,21,23'>1</vcpu> 
<vcpu cpuset='1,3,5,7,9,11,13,15,17,19,21,23'>2</vcpu> 
<vcpu cpuset='1,3,5,7,9,11,13,15,17,19,21,23'>3</vcpu> 
<vcpu cpuset='1,3,5,7,9,11,13,15,17,19,21,23'>4</vcpu> 

So you don't need the NUMA info from "virsh capabilities".

We now split up our hosts by the amount of NUMA-nodes to prevent
performance-degradation and swapping.

-- 
libvirt not recognizing NUMA architecture
https://bugs.launchpad.net/bugs/614322
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to libvirt in ubuntu.