[Bug 1927519] Re: Mitigate libvirt: error : unable to set AppArmor profile 'libvirt-<vm-uuid>' for '/usr/bin/kvm-spice': No such file or directory

Tue Nov 23 11:28:35 UTC 2021

root at f:~# apt upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  command-not-found curl libcurl3-gnutls libcurl4 libvirt-clients libvirt-daemon libvirt-daemon-driver-qemu libvirt-daemon-driver-storage-rbd libvirt-daemon-system
  libvirt-daemon-system-systemd libvirt0 libx11-6 libx11-data python3-commandnotfound secureboot-db sosreport wget
17 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 4868 kB of archives.
After this operation, 93.2 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 command-not-found all 20.04.5 [5244 B]
Get:2 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 python3-commandnotfound all 20.04.5 [10.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libx11-data all 2:1.6.9-2ubuntu1.3 [113 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libx11-6 amd64 2:1.6.9-2ubuntu1.3 [577 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 wget amd64 1.20.3-1ubuntu2 [348 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 curl amd64 7.68.0-1ubuntu2.8 [161 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.8 [234 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libcurl3-gnutls amd64 7.68.0-1ubuntu2.8 [233 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libvirt-daemon-driver-storage-rbd amd64 6.0.0-0ubuntu8.15 [28.3 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libvirt-daemon-driver-qemu amd64 6.0.0-0ubuntu8.15 [605 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libvirt-daemon-system amd64 6.0.0-0ubuntu8.15 [67.8 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libvirt-clients amd64 6.0.0-0ubuntu8.15 [344 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libvirt0 amd64 6.0.0-0ubuntu8.15 [1450 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libvirt-daemon amd64 6.0.0-0ubuntu8.15 [404 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 libvirt-daemon-system-systemd amd64 6.0.0-0ubuntu8.15 [8292 B]
Get:16 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 secureboot-db amd64 1.6~20.04.1 [14.2 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 sosreport amd64 4.2-1ubuntu0.20.04.1 [265 kB]
Fetched 4868 kB in 1s (6355 kB/s)    
Preconfiguring packages ...
(Reading database ... 48075 files and directories currently installed.)
Preparing to unpack .../00-command-not-found_20.04.5_all.deb ...
Unpacking command-not-found (20.04.5) over (20.04.4) ...
Preparing to unpack .../01-python3-commandnotfound_20.04.5_all.deb ...
Unpacking python3-commandnotfound (20.04.5) over (20.04.4) ...
Preparing to unpack .../02-libx11-data_2%3a1.6.9-2ubuntu1.3_all.deb ...
Unpacking libx11-data (2:1.6.9-2ubuntu1.3) over (2:1.6.9-2ubuntu1.2) ...
Preparing to unpack .../03-libx11-6_2%3a1.6.9-2ubuntu1.3_amd64.deb ...
Unpacking libx11-6:amd64 (2:1.6.9-2ubuntu1.3) over (2:1.6.9-2ubuntu1.2) ...
Preparing to unpack .../04-wget_1.20.3-1ubuntu2_amd64.deb ...
Unpacking wget (1.20.3-1ubuntu2) over (1.20.3-1ubuntu1) ...
Preparing to unpack .../05-curl_7.68.0-1ubuntu2.8_amd64.deb ...
Unpacking curl (7.68.0-1ubuntu2.8) over (7.68.0-1ubuntu2.7) ...
Preparing to unpack .../06-libcurl4_7.68.0-1ubuntu2.8_amd64.deb ...
Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.8) over (7.68.0-1ubuntu2.7) ...
Preparing to unpack .../07-libcurl3-gnutls_7.68.0-1ubuntu2.8_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.8) over (7.68.0-1ubuntu2.7) ...
Preparing to unpack .../08-libvirt-daemon-driver-storage-rbd_6.0.0-0ubuntu8.15_amd64.deb ...
Unpacking libvirt-daemon-driver-storage-rbd (6.0.0-0ubuntu8.15) over (6.0.0-0ubuntu8.15~focalppa1) ...
Preparing to unpack .../09-libvirt-daemon-driver-qemu_6.0.0-0ubuntu8.15_amd64.deb ...
Unpacking libvirt-daemon-driver-qemu (6.0.0-0ubuntu8.15) over (6.0.0-0ubuntu8.15~focalppa1) ...
Preparing to unpack .../10-libvirt-daemon-system_6.0.0-0ubuntu8.15_amd64.deb ...
Unpacking libvirt-daemon-system (6.0.0-0ubuntu8.15) over (6.0.0-0ubuntu8.15~focalppa1) ...
Preparing to unpack .../11-libvirt-clients_6.0.0-0ubuntu8.15_amd64.deb ...
Unpacking libvirt-clients (6.0.0-0ubuntu8.15) over (6.0.0-0ubuntu8.15~focalppa1) ...
Preparing to unpack .../12-libvirt0_6.0.0-0ubuntu8.15_amd64.deb ...
Unpacking libvirt0:amd64 (6.0.0-0ubuntu8.15) over (6.0.0-0ubuntu8.15~focalppa1) ...
Preparing to unpack .../13-libvirt-daemon_6.0.0-0ubuntu8.15_amd64.deb ...
Unpacking libvirt-daemon (6.0.0-0ubuntu8.15) over (6.0.0-0ubuntu8.15~focalppa1) ...
Preparing to unpack .../14-libvirt-daemon-system-systemd_6.0.0-0ubuntu8.15_amd64.deb ...
Unpacking libvirt-daemon-system-systemd (6.0.0-0ubuntu8.15) over (6.0.0-0ubuntu8.15~focalppa1) ...
Preparing to unpack .../15-secureboot-db_1.6~20.04.1_amd64.deb ...
Unpacking secureboot-db (1.6~20.04.1) over (1.5) ...
Preparing to unpack .../16-sosreport_4.2-1ubuntu0.20.04.1_amd64.deb ...
Unpacking sosreport (4.2-1ubuntu0.20.04.1) over (4.1-1ubuntu0.20.04.3) ...
Setting up wget (1.20.3-1ubuntu2) ...
Setting up secureboot-db (1.6~20.04.1) ...
Setting up libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.8) ...
Setting up libx11-data (2:1.6.9-2ubuntu1.3) ...
Setting up python3-commandnotfound (20.04.5) ...
Setting up libvirt-daemon-system-systemd (6.0.0-0ubuntu8.15) ...
Setting up sosreport (4.2-1ubuntu0.20.04.1) ...
Setting up libvirt0:amd64 (6.0.0-0ubuntu8.15) ...
Setting up libcurl4:amd64 (7.68.0-1ubuntu2.8) ...
Setting up libx11-6:amd64 (2:1.6.9-2ubuntu1.3) ...
Setting up curl (7.68.0-1ubuntu2.8) ...
Setting up libvirt-clients (6.0.0-0ubuntu8.15) ...
Setting up command-not-found (20.04.5) ...
Setting up libvirt-daemon-driver-qemu (6.0.0-0ubuntu8.15) ...
Setting up libvirt-daemon (6.0.0-0ubuntu8.15) ...
Setting up libvirt-daemon-driver-storage-rbd (6.0.0-0ubuntu8.15) ...
Setting up libvirt-daemon-system (6.0.0-0ubuntu8.15) ...
Job failed. See "journalctl -xe" for details.
virtlockd.service is a disabled or a static unit, not starting it.
virtlogd.service is a disabled or a static unit, not starting it.
Setting up libvirt-daemon dnsmasq configuration.
Processing triggers for install-info (6.7.0.dfsg.2-5) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for man-db (2.9.1-1) ...

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1927519

Title:
  Mitigate libvirt: error : unable to set AppArmor profile 'libvirt-<vm-
  uuid>' for '/usr/bin/kvm-spice': No such file or directory

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive queens series:
  New
Status in Ubuntu Cloud Archive stein series:
  New
Status in Ubuntu Cloud Archive ussuri series:
  New
Status in libvirt package in Ubuntu:
  Fix Committed
Status in libvirt source package in Bionic:
  Fix Committed
Status in libvirt source package in Focal:
  Fix Committed
Status in libvirt source package in Hirsute:
  Fix Committed
Status in libvirt source package in Impish:
  Fix Committed
Status in libvirt source package in Jammy:
  Fix Committed

Bug description:
  [Impact]

  Some times libvirt fails to start a vm with the following error :
  libvirt:  error : unable to set AppArmor profile 'libvirt-b05b297f-952f-42d6-b04e-f9a13767db54' for '/usr/bin/kvm-spice': No such file or directory

  This happens because for some reason the apparmor profile for the
  guest /etc/apparmor.d/libvirt/libvirt-<vm-uuid> has 0 size.

  We do not now why the above file gets truncated to begin with and
  therefore we do not know the root cause to fix it there. But the
  condition is easy to detect and 100% broken, so we can detect and
  recreate the file in those cases.

  [Test case]

  To reproduce this behaviour, create a vm and stop it, note the uuid.

  For example:
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
  $ uvt-kvm create --password=ubuntu f release=focal arch=amd64 label=daily
  $ virsh dominfo f
  ...
  Security label: libvirt-1ceac8db-c1e9-40b2-8ada-a60349454fc1 (enforcing)
  $ virsh shutdown f

  Then make the start apparmor profile an empty file.
  On Bionic/Focal that file will be non-existant by default (cleaned on guest stop), on Hirsute/Impish it will be around for admin-edit but with content. Therefore "touch" isn't enough every time, instead really put nothing to it as that is how the real issue looks like).

  $ cat /dev/null | sudo tee /etc/apparmor.d/libvirt/libvirt-1ceac8db-c1e9-40b2-8ada-a60349454fc1
  # ensure it is size zero
  $ ll /etc/apparmor.d/libvirt/libvirt-1ceac8db-c1e9-40b2-8ada-a60349454fc1
  -rw-r--r-- 1 root root 0 Nov 18 09:01 /etc/apparmor.d/libvirt/libvirt-1ceac8db-c1e9-40b2-8ada-a60349454fc1

  Next try to start the vm, it will try to use the file it found
  (instead of creating a new one as it would when non-existing) and will
  fail doing so:

  $ virsh start f
  error: internal error: Process exited prior to exec: libvirt:  error : unable to set AppArmor profile 'libvirt-1ceac8db-c1e9-40b2-8ada-a60349454fc1' for '/usr/bin/qemu-system-x86_64': No such file or directory

  To manually be able to start the vm again just delete the libvirt-<vm-uuid> file.
  With the fix applied libvirt will recreate the file and guest start works again.

  In addition (independent to the case) I'll run a set of common regression tests against <release>-proposed which didn't run for a while and would make us also spot if anything other slipped in from different places (like the dwarves hiccup we had). Since testing shall be on "the real build" and resources are limited I'd this time do so only on the builds in proposed.

  [Regression Potential]

  The new code is only active when the size of the file is zero which is
  a 100% guarantee that the guest is broken and won't start.
  Nevertheless if we made a mistake in the fix the area (of the many
  things libvirt does) to look at is the generating and usage of
  apparmor profiles.

  [Other]

  Similar reported bug : https://bugs.debian.org/cgi-
  bin/bugreport.cgi?bug=890084

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1927519/+subscriptions