[Bug 614322] Re: libvirt not recognizing NUMA architecture
EAB
614322 at bugs.launchpad.net
Sun Nov 7 14:43:19 GMT 2010
I can confirm this.
Most of our hosts have 64GB RAM and 2 Intel Nehalem hexa-cores.
With 3 VM's using 16GB RAM each.
I noticed performance degredation and the host also swapped out a lot of memory. The swapping degraded performance dramatically.
vm.swappiness=0 in /etc/sysctl.conf did not help.
It seems that NUMA on Intel CPU's can be expensive because RAM needs to
be transfered from other nodes. With only 1 node (socket) there is no
problem. With 2 or more nodes you see slower.
Without the capabilities you can prevent this behavior by pinning the vcpu's.
You should spread you VM's over the available nodes:
numactl --hardware | grep "node 0 cpus"
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22
Your XML should contain something like this:
<vcpu cpuset='0,2,4,6,8,10,12,14,16,18,20,22'>1</vcpu>
<vcpu cpuset='0,2,4,6,8,10,12,14,16,18,20,22'>2</vcpu>
<vcpu cpuset='0,2,4,6,8,10,12,14,16,18,20,22'>3</vcpu>
<vcpu cpuset='0,2,4,6,8,10,12,14,16,18,20,22'>4</vcpu>
The next VM should use they other node.
numactl --hardware | grep "node 1 cpus"
node 0 cpus: 1 3 5 7 9 11 13 15 17 19 21 23
<vcpu cpuset='1,3,5,7,9,11,13,15,17,19,21,23'>1</vcpu>
<vcpu cpuset='1,3,5,7,9,11,13,15,17,19,21,23'>2</vcpu>
<vcpu cpuset='1,3,5,7,9,11,13,15,17,19,21,23'>3</vcpu>
<vcpu cpuset='1,3,5,7,9,11,13,15,17,19,21,23'>4</vcpu>
So you don't need the NUMA info from "virsh capabilities".
We now split up our hosts by the amount of NUMA-nodes to prevent
performance-degradation and swapping.
--
libvirt not recognizing NUMA architecture
https://bugs.launchpad.net/bugs/614322
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to libvirt in ubuntu.
More information about the Ubuntu-server-bugs
mailing list