device-mapper: core: unable to allocate device, out of memory

Fri Feb 2 13:43:34 UTC 2018

Hi,

On a fresh install of Ubuntu 17.10 which I am using as a Xen hypervisor to host virtual machines I encountered a problem where I can't create any LVM2 logical volumes anymore as it seems to strangely run out of memory.

On my Xen hypervisor I have assigned 2 GB of memory a pinned 2 CPU cores reserved for the hypervisor and have currently nothing running, not even a VM. Here is the problem I encounter:

$ sudo lvcreate  -L 10G --name test2 xenvg0
  device-mapper: create ioctl on xenvg0-test2LVM-21isjzgjLTDUDPhMEVXeW2LejUbLzRh9Yz0SB6YnREm0TT3sQ26jGdkJJnMcc32F failed: No such device or address
  Failed to activate new LV.

The current memory consumption is the following:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1920         120          18          10        1782        1757
Swap:          2047           0        2047

When running that command the following gets logged onto the kernel console:

[185140.173774] lvcreate: page allocation failure: order:4, mode:0x160c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null)
[185140.173779] lvcreate cpuset=/ mems_allowed=0
[185140.173784] CPU: 0 PID: 23440 Comm: lvcreate Tainted: G        W       4.13.0-32-generic #35-Ubuntu
[185140.173785] Hardware name: Lenovo ThinkSystem SR630 -[7X02CTO1WW]-/-[7X02CTO1WW]-, BIOS -[IVE112I-1.01]- 08/11/2017
[185140.173786] Call Trace:
[185140.173794]  dump_stack+0x63/0x8b
[185140.173799]  warn_alloc+0x114/0x1c0
[185140.173802]  ? __alloc_pages_direct_compact+0x51/0x100
[185140.173804]  __alloc_pages_slowpath+0xd78/0xe30
[185140.173806]  ? get_page_from_freelist+0x278/0xb20
[185140.173809]  ? __follow_mount_rcu.isra.22+0x68/0xe0
[185140.173811]  __alloc_pages_nodemask+0x25d/0x280
[185140.173815]  ? dev_wait+0xd0/0xd0
[185140.173818]  kmalloc_large_node+0x3b/0x60
[185140.173819]  __kmalloc_node+0x204/0x2a0
[185140.173821]  ? dev_wait+0xd0/0xd0
[185140.173824]  dm_create+0x62/0x530
[185140.173825]  ? dev_wait+0xd0/0xd0
[185140.173826]  dev_create+0x6a/0x330
[185140.173831]  ? security_capable+0x48/0x60
[185140.173833]  ? dev_wait+0xd0/0xd0
[185140.173834]  ctl_ioctl+0x1ed/0x570
[185140.173837]  dm_ctl_ioctl+0xe/0x20
[185140.173840]  do_vfs_ioctl+0xa5/0x610
[185140.173843]  ? entry_SYSCALL_64_after_hwframe+0x118/0x168
[185140.173845]  ? entry_SYSCALL_64_after_hwframe+0x111/0x168
[185140.173846]  ? entry_SYSCALL_64_after_hwframe+0x10a/0x168
[185140.173848]  ? entry_SYSCALL_64_after_hwframe+0x103/0x168
[185140.173849]  ? entry_SYSCALL_64_after_hwframe+0xfc/0x168
[185140.173850]  ? entry_SYSCALL_64_after_hwframe+0xf5/0x168
[185140.173852]  ? entry_SYSCALL_64_after_hwframe+0xee/0x168
[185140.173853]  ? entry_SYSCALL_64_after_hwframe+0xe7/0x168
[185140.173854]  ? entry_SYSCALL_64_after_hwframe+0xe0/0x168
[185140.173856]  SyS_ioctl+0x79/0x90
[185140.173858]  ? entry_SYSCALL_64_after_hwframe+0xa1/0x168
[185140.173859]  entry_SYSCALL_64_fastpath+0x33/0xa3
[185140.173861] RIP: 0033:0x7efc38207ef7
[185140.173862] RSP: 002b:00007fffe5234788 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[185140.173863] RAX: ffffffffffffffda RBX: 000055c4da2eb990 RCX: 00007efc38207ef7
[185140.173864] RDX: 000055c4db48fa50 RSI: 00000000c138fd03 RDI: 0000000000000004
[185140.173865] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fffe52345f0
[185140.173865] R10: 00007efc389669f3 R11: 0000000000000246 R12: 0000000000000000
[185140.173866] R13: 000055c4db48fa80 R14: 000055c4db48fa50 R15: 000055c4db47e2e0
[185140.173868] Mem-Info:
[185140.173870] active_anon:1564 inactive_anon:10524 isolated_anon:0
                 active_file:41318 inactive_file:382992 isolated_file:0
                 unevictable:4127 dirty:7293 writeback:0 unstable:0
                 slab_reclaimable:25020 slab_unreclaimable:4452
                 mapped:8607 shmem:2732 pagetables:1628 bounce:0
                 free:4536 free_pcp:0 free_cma:0
[185140.173873] Node 0 active_anon:6256kB inactive_anon:42096kB active_file:165272kB inactive_file:1531968kB unevictable:16508kB isolated(anon):0kB isolated(file):0kB mapped:34428kB dirty:29172kB writeback:0kB shmem:10928kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[185140.173873] Node 0 DMA free:7644kB min:44kB low:56kB high:68kB active_anon:0kB inactive_anon:4kB active_file:244kB inactive_file:7508kB unevictable:0kB writepending:12kB present:15996kB managed:15912kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[185140.173876] lowmem_reserve[]: 0 1900 1900 1900 1900
[185140.173878] Node 0 DMA32 free:6268kB min:5540kB low:7484kB high:9428kB active_anon:6256kB inactive_anon:42092kB active_file:165028kB inactive_file:1524460kB unevictable:16508kB writepending:29180kB present:2081152kB managed:1946092kB mlocked:16508kB kernel_stack:2144kB pagetables:6512kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[185140.173881] lowmem_reserve[]: 0 0 0 0 0
[185140.173883] Node 0 Normal free:4232kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:131072kB managed:4232kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[185140.173885] lowmem_reserve[]: 0 0 0 0 0
[185140.173887] Node 0 DMA: 1*4kB (M) 3*8kB (UE) 2*16kB (UM) 3*32kB (UME) 3*64kB (U) 3*128kB (UM) 1*256kB (M) 1*512kB (E) 2*1024kB (UE) 2*2048kB (ME) 0*4096kB = 7644kB
[185140.173896] Node 0 DMA32: 146*4kB (UME) 95*8kB (UM) 69*16kB (UME) 56*32kB (UME) 32*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6288kB
[185140.173902] Node 0 Normal: 0*4kB 1*8kB (M) 0*16kB 0*32kB 0*64kB 1*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (M) = 4232kB
[185140.173908] 428368 total pagecache pages
[185140.173910] 0 pages in swap cache
[185140.173910] Swap cache stats: add 0, delete 0, find 0/0
[185140.173911] Free swap  = 2097148kB
[185140.173911] Total swap = 2097148kB
[185140.173912] 557055 pages RAM
[185140.173912] 0 pages HighMem/MovableOnly
[185140.173913] 65496 pages reserved
[185140.173913] 0 pages cma reserved
[185140.173913] 0 pages hwpoisoned
[185140.173914] device-mapper: core: unable to allocate device, out of memory.

Does anyone have any idea what could be going wrong here?

I can't believe that I need more RAM on the hypervisor in order to create logical volumes, that sounds crazy.

Thanks for the insights.

Regards,
M.