[Bug 1674481] Re: memory overhead of qemu-kvm with ceph rbd and ram-allocation-ratio=0.9 leads to memory starvation

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: qemu (Ubuntu)
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1674481

Title:
  memory overhead of qemu-kvm with ceph rbd and ram-allocation-ratio=0.9
  leads to memory starvation

Status in OpenStack nova-cloud-controller charm:
  Invalid
Status in ceph package in Ubuntu:
  Incomplete
Status in qemu package in Ubuntu:
  Confirmed

Bug description:
  We have observed up to 20% memory overhead on several 18GB nova
  instances packed per node taking up to 22GB resident memory when fully
  utilized.  Our standard ram-allocation-ratio is 0.9  and we have 5120
  MB reserved-host-memory configured on the nova-compute charm.

  The nodes have 512 GB ram and were experiencing less than 8GB free ram
  with 8GB swap utilized.  High consumption CICD environment with ceph
  ephemeral disk.

  We have since worked around memory starvation by adding nodes and
  reducing ram_allocation_ratio to 0.7.

  Does this resident memory overhead seem abnormally high?  Is there
  anything in the below qemu process that may be causing this that we
  need to account for in the ram-allocation-ratio settings?

  Specs:
  Xenial series
  nova-cloud-controller 13.1.2 charm rev. 503
  nova-compute 13.1.2 charm rev. 135

  Note the -m 18432(MB) argument and 22049 MB RSS of this typical high-
  use instance's process:

  USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  libvirt+ 2020854  208  4.2 49713524 22579016 ?   Sl   19:06 235:28 /usr/bin/qemu-system-x86_64 -name instance-XXXXXXXX -S -machine pc-i440fx-xenial
  accel=kvm
  usb=off -cpu Haswell-noTSX -m 18432 -realtime mlock=off -smp 8
  sockets=8
  cores=1
  threads=1 -uuid XXXXXXXX -smbios type=1
  manufacturer=OpenStack Foundation
  product=OpenStack Nova
  version=13.1.2
  serial=XXXXXXXXXXXX
  uuid=XXXXXXXXX
  family=Virtual Machine -no-user-config -nodefaults -chardev socket
  id=charmonitor
  path=/var/lib/libvirt/qemu/domain-instance-000173b7/monitor.sock
  server
  nowait -mon chardev=charmonitor
  id=monitor
  mode=control -rtc base=utc
  driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci
  id=usb
  bus=pci.0
  addr=0x1.0x2 -drive file=rbd:nova/XXXXXXX-23e3-40c0-9038-3dd837e5b1a3_disk:id=nova-compute:key=XXXXXXXXX==:auth_supported=cephx\;none:mon_host=X.Y.Z.A\:6789\;1X.Y.Z.B\:6789\;X.Y.Z.C\:6789
  format=raw
  if=none
  id=drive-virtio-disk0
  cache=none -device virtio-blk-pci
  scsi=off
  bus=pci.0
  addr=0x4
  drive=drive-virtio-disk0
  id=virtio-disk0
  bootindex=1 -netdev tap
  fd=26
  id=hostnet0
  vhost=on
  vhostfd=34 -device virtio-net-pci
  netdev=hostnet0
  id=net0
  mac=XX:XX:XX:XX:XX:XX
  bus=pci.0
  addr=0x3 -chardev file
  id=charserial0
  path=/var/lib/nova/instances/29ce4bc7-23e3-40c0-9038-3dd837e5b1a3/console.log -device isa-serial
  chardev=charserial0
  id=serial0 -chardev pty
  id=charserial1 -device isa-serial
  chardev=charserial1
  id=serial1 -device usb-tablet
  id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga
  id=video0
  bus=pci.0
  addr=0x2 -device virtio-balloon-pci
  id=balloon0
  bus=pci.0
  addr=0x5 -msg timestamp=on

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1674481/+subscriptions