[Bug 1446177] [NEW] Nodeinfo returns wrong NUMA topology / bad virtualization performance
FliTTi
m.flittner at gmx.net
Mon Apr 20 11:23:06 UTC 2015
Public bug reported:
1)
Description: Ubuntu 14.04.2 LTS
Release: 14.0
2)
libvirt-bin:
Installed: 1.2.2-0ubuntu13.1.10
Candidate: 1.2.2-0ubuntu13.1.10
Version table:
*** 1.2.2-0ubuntu13.1.10 0
500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
100 /var/lib/dpkg/status
1.2.2-0ubuntu13.1.7 0
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
1.2.2-0ubuntu13 0
500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
4+5)
"virsh nodeinfo" shows wrong NUMA topology:
:~$ virsh nodeinfo
CPU model: x86_64
CPU(s): 64
CPU frequency: 1400 MHz
CPU socket(s): 1
Core(s) per socket: 64
Thread(s) per core: 1
NUMA cell(s): 1
Memory size: 528376144 KiB
but it should be "NUMA cell(s): 8".
I'm running a 3.13.0-48-generic equipped with _four_ physical AMD
Opteron 6282 SE to a Supermicro H8QG6-F. libvirt-bin version:
1.2.2-0ubuntu13.1.10
NUMA ctl gives me eight cells:
:~$ numactl -H
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64431 MB
node 0 free: 58505 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 64510 MB
node 1 free: 45068 MB
node 2 cpus: 16 17 18 19 20 21 22 23
node 2 size: 64510 MB
node 2 free: 62910 MB
node 3 cpus: 24 25 26 27 28 29 30 31
node 3 size: 64510 MB
node 3 free: 20460 MB
node 4 cpus: 32 33 34 35 36 37 38 39
node 4 size: 64510 MB
node 4 free: 63512 MB
node 5 cpus: 40 41 42 43 44 45 46 47
node 5 size: 64510 MB
node 5 free: 52723 MB
node 6 cpus: 48 49 50 51 52 53 54 55
node 6 size: 64510 MB
node 6 free: 64084 MB
node 7 cpus: 56 57 58 59 60 61 62 63
node 7 size: 64494 MB
node 7 free: 62013 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 16 16 22 16 22 16 22
1: 16 10 22 16 22 16 22 16
2: 16 22 10 16 16 22 16 22
3: 22 16 16 10 22 16 22 16
4: 16 22 16 22 10 16 16 22
5: 22 16 22 16 16 10 22 16
6: 16 22 16 22 16 22 10 16
7: 22 16 22 16 22 16 16 10
As well as lstopo is resulting in eight NUMA cells (see attached
picture).
But "virsh nodeinfo" shows only only NUMA cell with one socket with 64
CPUs (single threaded). Performance of virtualized machines on this host
is very bad.
I think it has something to do with the identical core_ids per physical
socket and module.
:~$ cat /proc/cpuinfo | egrep "processor|physical id|siblings|core id|cpu cores"
processor : 0
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
processor : 1
physical id : 0
siblings : 16
core id : 1
cpu cores : 8
processor : 2
physical id : 0
siblings : 16
core id : 2
cpu cores : 8
processor : 3
physical id : 0
siblings : 16
core id : 3
cpu cores : 8
processor : 4
physical id : 0
siblings : 16
core id : 4
cpu cores : 8
processor : 5
physical id : 0
siblings : 16
core id : 5
cpu cores : 8
processor : 6
physical id : 0
siblings : 16
core id : 6
cpu cores : 8
processor : 7
physical id : 0
siblings : 16
core id : 7
cpu cores : 8
processor : 8
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
processor : 9
physical id : 0
siblings : 16
core id : 1
cpu cores : 8
processor : 10
physical id : 0
siblings : 16
core id : 2
cpu cores : 8
processor : 11
physical id : 0
siblings : 16
core id : 3
cpu cores : 8
processor : 12
physical id : 0
siblings : 16
core id : 4
cpu cores : 8
processor : 13
physical id : 0
siblings : 16
core id : 5
cpu cores : 8
processor : 14
physical id : 0
siblings : 16
core id : 6
cpu cores : 8
processor : 15
physical id : 0
siblings : 16
core id : 7
cpu cores : 8
processor : 16
physical id : 1
siblings : 16
core id : 0
cpu cores : 8
processor : 17
physical id : 1
siblings : 16
core id : 1
cpu cores : 8
processor : 18
physical id : 1
siblings : 16
core id : 2
cpu cores : 8
processor : 19
physical id : 1
siblings : 16
core id : 3
cpu cores : 8
processor : 20
physical id : 1
siblings : 16
core id : 4
cpu cores : 8
processor : 21
physical id : 1
siblings : 16
core id : 5
cpu cores : 8
processor : 22
physical id : 1
siblings : 16
core id : 6
cpu cores : 8
processor : 23
physical id : 1
siblings : 16
core id : 7
cpu cores : 8
processor : 24
physical id : 1
siblings : 16
core id : 0
cpu cores : 8
processor : 25
physical id : 1
siblings : 16
core id : 1
cpu cores : 8
processor : 26
physical id : 1
siblings : 16
core id : 2
cpu cores : 8
processor : 27
physical id : 1
siblings : 16
core id : 3
cpu cores : 8
processor : 28
physical id : 1
siblings : 16
core id : 4
cpu cores : 8
processor : 29
physical id : 1
siblings : 16
core id : 5
cpu cores : 8
processor : 30
physical id : 1
siblings : 16
core id : 6
cpu cores : 8
processor : 31
physical id : 1
siblings : 16
core id : 7
cpu cores : 8
processor : 32
physical id : 2
siblings : 16
core id : 0
cpu cores : 8
processor : 33
physical id : 2
siblings : 16
core id : 1
cpu cores : 8
processor : 34
physical id : 2
siblings : 16
core id : 2
cpu cores : 8
processor : 35
physical id : 2
siblings : 16
core id : 3
cpu cores : 8
processor : 36
physical id : 2
siblings : 16
core id : 4
cpu cores : 8
processor : 37
physical id : 2
siblings : 16
core id : 5
cpu cores : 8
processor : 38
physical id : 2
siblings : 16
core id : 6
cpu cores : 8
processor : 39
physical id : 2
siblings : 16
core id : 7
cpu cores : 8
processor : 40
physical id : 2
siblings : 16
core id : 0
cpu cores : 8
processor : 41
physical id : 2
siblings : 16
core id : 1
cpu cores : 8
processor : 42
physical id : 2
siblings : 16
core id : 2
cpu cores : 8
processor : 43
physical id : 2
siblings : 16
core id : 3
cpu cores : 8
processor : 44
physical id : 2
siblings : 16
core id : 4
cpu cores : 8
processor : 45
physical id : 2
siblings : 16
core id : 5
cpu cores : 8
processor : 46
physical id : 2
siblings : 16
core id : 6
cpu cores : 8
processor : 47
physical id : 2
siblings : 16
core id : 7
cpu cores : 8
processor : 48
physical id : 3
siblings : 16
core id : 0
cpu cores : 8
processor : 49
physical id : 3
siblings : 16
core id : 1
cpu cores : 8
processor : 50
physical id : 3
siblings : 16
core id : 2
cpu cores : 8
processor : 51
physical id : 3
siblings : 16
core id : 3
cpu cores : 8
processor : 52
physical id : 3
siblings : 16
core id : 4
cpu cores : 8
processor : 53
physical id : 3
siblings : 16
core id : 5
cpu cores : 8
processor : 54
physical id : 3
siblings : 16
core id : 6
cpu cores : 8
processor : 55
physical id : 3
siblings : 16
core id : 7
cpu cores : 8
processor : 56
physical id : 3
siblings : 16
core id : 0
cpu cores : 8
processor : 57
physical id : 3
siblings : 16
core id : 1
cpu cores : 8
processor : 58
physical id : 3
siblings : 16
core id : 2
cpu cores : 8
processor : 59
physical id : 3
siblings : 16
core id : 3
cpu cores : 8
processor : 60
physical id : 3
siblings : 16
core id : 4
cpu cores : 8
processor : 61
physical id : 3
siblings : 16
core id : 5
cpu cores : 8
processor : 62
physical id : 3
siblings : 16
core id : 6
cpu cores : 8
processor : 63
physical id : 3
siblings : 16
core id : 7
cpu cores : 8
Thanks!
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: libvirt-bin 1.2.2-0ubuntu13.1.10
ProcVersionSignature: Ubuntu 3.13.0-48.80-generic 3.13.11-ckt16
Uname: Linux 3.13.0-48-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
Date: Mon Apr 20 13:04:31 2015
InstallationDate: Installed on 2012-12-07 (864 days ago)
InstallationMedia: Ubuntu-Server 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120817.3)
KernLog:
SourcePackage: libvirt
UpgradeStatus: Upgraded to trusty on 2014-07-07 (286 days ago)
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission denied: '/etc/libvirt/qemu.conf']
** Affects: libvirt (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug trusty
** Attachment added: "lstopo output"
https://bugs.launchpad.net/bugs/1446177/+attachment/4379935/+files/arch.eps
** Description changed:
- 1)
+ 1)
Description: Ubuntu 14.04.2 LTS
Release: 14.0
2)
libvirt-bin:
- Installed: 1.2.2-0ubuntu13.1.10
- Candidate: 1.2.2-0ubuntu13.1.10
- Version table:
- *** 1.2.2-0ubuntu13.1.10 0
- 500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
- 100 /var/lib/dpkg/status
- 1.2.2-0ubuntu13.1.7 0
- 500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
- 1.2.2-0ubuntu13 0
- 500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
+ Installed: 1.2.2-0ubuntu13.1.10
+ Candidate: 1.2.2-0ubuntu13.1.10
+ Version table:
+ *** 1.2.2-0ubuntu13.1.10 0
+ 500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
+ 100 /var/lib/dpkg/status
+ 1.2.2-0ubuntu13.1.7 0
+ 500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
+ 1.2.2-0ubuntu13 0
+ 500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
4+5)
"virsh nodeinfo" shows wrong NUMA topology:
:~$ virsh nodeinfo
CPU model: x86_64
CPU(s): 64
CPU frequency: 1400 MHz
CPU socket(s): 1
Core(s) per socket: 64
Thread(s) per core: 1
NUMA cell(s): 1
Memory size: 528376144 KiB
but it should be "NUMA cell(s): 8".
I'm running a 3.13.0-48-generic equipped with _four_ physical AMD
Opteron 6282 SE to a Supermicro H8QG6-F. libvirt-bin version:
1.2.2-0ubuntu13.1.10
NUMA ctl gives me eight cells:
:~$ numactl -H
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64431 MB
node 0 free: 58505 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 64510 MB
node 1 free: 45068 MB
node 2 cpus: 16 17 18 19 20 21 22 23
node 2 size: 64510 MB
node 2 free: 62910 MB
node 3 cpus: 24 25 26 27 28 29 30 31
node 3 size: 64510 MB
node 3 free: 20460 MB
node 4 cpus: 32 33 34 35 36 37 38 39
node 4 size: 64510 MB
node 4 free: 63512 MB
node 5 cpus: 40 41 42 43 44 45 46 47
node 5 size: 64510 MB
node 5 free: 52723 MB
node 6 cpus: 48 49 50 51 52 53 54 55
node 6 size: 64510 MB
node 6 free: 64084 MB
node 7 cpus: 56 57 58 59 60 61 62 63
node 7 size: 64494 MB
node 7 free: 62013 MB
node distances:
- node 0 1 2 3 4 5 6 7
- 0: 10 16 16 22 16 22 16 22
- 1: 16 10 22 16 22 16 22 16
- 2: 16 22 10 16 16 22 16 22
- 3: 22 16 16 10 22 16 22 16
- 4: 16 22 16 22 10 16 16 22
- 5: 22 16 22 16 16 10 22 16
- 6: 16 22 16 22 16 22 10 16
- 7: 22 16 22 16 22 16 16 10
+ node 0 1 2 3 4 5 6 7
+ 0: 10 16 16 22 16 22 16 22
+ 1: 16 10 22 16 22 16 22 16
+ 2: 16 22 10 16 16 22 16 22
+ 3: 22 16 16 10 22 16 22 16
+ 4: 16 22 16 22 10 16 16 22
+ 5: 22 16 22 16 16 10 22 16
+ 6: 16 22 16 22 16 22 10 16
+ 7: 22 16 22 16 22 16 16 10
As well as lstopo is resulting in eight NUMA cells (see attached
picture).
But "virsh nodeinfo" shows only only NUMA cell with one socket with 64
- CPUs (single threaded). Performance of virtualized machines one this
- host is very bad.
+ CPUs (single threaded). Performance of virtualized machines on this host
+ is very bad.
I think it has something to do with the identical core_ids per physical
socket and module.
:~$ cat /proc/cpuinfo | egrep "processor|physical id|siblings|core id|cpu cores"
processor : 0
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
processor : 1
physical id : 0
siblings : 16
core id : 1
cpu cores : 8
processor : 2
physical id : 0
siblings : 16
core id : 2
cpu cores : 8
processor : 3
physical id : 0
siblings : 16
core id : 3
cpu cores : 8
processor : 4
physical id : 0
siblings : 16
core id : 4
cpu cores : 8
processor : 5
physical id : 0
siblings : 16
core id : 5
cpu cores : 8
processor : 6
physical id : 0
siblings : 16
core id : 6
cpu cores : 8
processor : 7
physical id : 0
siblings : 16
core id : 7
cpu cores : 8
processor : 8
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
processor : 9
physical id : 0
siblings : 16
core id : 1
cpu cores : 8
processor : 10
physical id : 0
siblings : 16
core id : 2
cpu cores : 8
processor : 11
physical id : 0
siblings : 16
core id : 3
cpu cores : 8
processor : 12
physical id : 0
siblings : 16
core id : 4
cpu cores : 8
processor : 13
physical id : 0
siblings : 16
core id : 5
cpu cores : 8
processor : 14
physical id : 0
siblings : 16
core id : 6
cpu cores : 8
processor : 15
physical id : 0
siblings : 16
core id : 7
cpu cores : 8
processor : 16
physical id : 1
siblings : 16
core id : 0
cpu cores : 8
processor : 17
physical id : 1
siblings : 16
core id : 1
cpu cores : 8
processor : 18
physical id : 1
siblings : 16
core id : 2
cpu cores : 8
processor : 19
physical id : 1
siblings : 16
core id : 3
cpu cores : 8
processor : 20
physical id : 1
siblings : 16
core id : 4
cpu cores : 8
processor : 21
physical id : 1
siblings : 16
core id : 5
cpu cores : 8
processor : 22
physical id : 1
siblings : 16
core id : 6
cpu cores : 8
processor : 23
physical id : 1
siblings : 16
core id : 7
cpu cores : 8
processor : 24
physical id : 1
siblings : 16
core id : 0
cpu cores : 8
processor : 25
physical id : 1
siblings : 16
core id : 1
cpu cores : 8
processor : 26
physical id : 1
siblings : 16
core id : 2
cpu cores : 8
processor : 27
physical id : 1
siblings : 16
core id : 3
cpu cores : 8
processor : 28
physical id : 1
siblings : 16
core id : 4
cpu cores : 8
processor : 29
physical id : 1
siblings : 16
core id : 5
cpu cores : 8
processor : 30
physical id : 1
siblings : 16
core id : 6
cpu cores : 8
processor : 31
physical id : 1
siblings : 16
core id : 7
cpu cores : 8
processor : 32
physical id : 2
siblings : 16
core id : 0
cpu cores : 8
processor : 33
physical id : 2
siblings : 16
core id : 1
cpu cores : 8
processor : 34
physical id : 2
siblings : 16
core id : 2
cpu cores : 8
processor : 35
physical id : 2
siblings : 16
core id : 3
cpu cores : 8
processor : 36
physical id : 2
siblings : 16
core id : 4
cpu cores : 8
processor : 37
physical id : 2
siblings : 16
core id : 5
cpu cores : 8
processor : 38
physical id : 2
siblings : 16
core id : 6
cpu cores : 8
processor : 39
physical id : 2
siblings : 16
core id : 7
cpu cores : 8
processor : 40
physical id : 2
siblings : 16
core id : 0
cpu cores : 8
processor : 41
physical id : 2
siblings : 16
core id : 1
cpu cores : 8
processor : 42
physical id : 2
siblings : 16
core id : 2
cpu cores : 8
processor : 43
physical id : 2
siblings : 16
core id : 3
cpu cores : 8
processor : 44
physical id : 2
siblings : 16
core id : 4
cpu cores : 8
processor : 45
physical id : 2
siblings : 16
core id : 5
cpu cores : 8
processor : 46
physical id : 2
siblings : 16
core id : 6
cpu cores : 8
processor : 47
physical id : 2
siblings : 16
core id : 7
cpu cores : 8
processor : 48
physical id : 3
siblings : 16
core id : 0
cpu cores : 8
processor : 49
physical id : 3
siblings : 16
core id : 1
cpu cores : 8
processor : 50
physical id : 3
siblings : 16
core id : 2
cpu cores : 8
processor : 51
physical id : 3
siblings : 16
core id : 3
cpu cores : 8
processor : 52
physical id : 3
siblings : 16
core id : 4
cpu cores : 8
processor : 53
physical id : 3
siblings : 16
core id : 5
cpu cores : 8
processor : 54
physical id : 3
siblings : 16
core id : 6
cpu cores : 8
processor : 55
physical id : 3
siblings : 16
core id : 7
cpu cores : 8
processor : 56
physical id : 3
siblings : 16
core id : 0
cpu cores : 8
processor : 57
physical id : 3
siblings : 16
core id : 1
cpu cores : 8
processor : 58
physical id : 3
siblings : 16
core id : 2
cpu cores : 8
processor : 59
physical id : 3
siblings : 16
core id : 3
cpu cores : 8
processor : 60
physical id : 3
siblings : 16
core id : 4
cpu cores : 8
processor : 61
physical id : 3
siblings : 16
core id : 5
cpu cores : 8
processor : 62
physical id : 3
siblings : 16
core id : 6
cpu cores : 8
processor : 63
physical id : 3
siblings : 16
core id : 7
cpu cores : 8
Thanks!
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: libvirt-bin 1.2.2-0ubuntu13.1.10
ProcVersionSignature: Ubuntu 3.13.0-48.80-generic 3.13.11-ckt16
Uname: Linux 3.13.0-48-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
Date: Mon Apr 20 13:04:31 2015
InstallationDate: Installed on 2012-12-07 (864 days ago)
InstallationMedia: Ubuntu-Server 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120817.3)
KernLog:
-
+
SourcePackage: libvirt
UpgradeStatus: Upgraded to trusty on 2014-07-07 (286 days ago)
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission denied: '/etc/libvirt/qemu.conf']
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to libvirt in Ubuntu.
https://bugs.launchpad.net/bugs/1446177
Title:
Nodeinfo returns wrong NUMA topology / bad virtualization performance
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1446177/+subscriptions
More information about the Ubuntu-server-bugs
mailing list