[Bug 1885269] Re: ussuri: unable to stop and start an instance post upgrade

[Alex Kavanagh]

Additional details from a compute unit:

on nova/compute/1 -- before stopping the unit, but after upgrading from
train->ussuri on bionic.  Note that the disk, console.log are owned
nova.nova.


root at juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184692
drwxr-x--- 2 nova nova      4096 Jun 25 17:46 ./
drwxr-x--- 5 nova nova      4096 Jun 25 18:21 ../
-rw-r----- 1 nova nova     31316 Jun 25 17:47 console.log
-rw-r----- 1 nova nova 189136896 Jun 26 10:17 disk
-rw-r----- 1 nova nova        79 Jun 25 17:46 disk.info
root at juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90

After stopping the instance, the disk is now owned root.root.  As there
is no world-readable perm, nova now can't restart the instance.

root at juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184948
drwxr-x--- 2 nova nova      4096 Jun 25 17:46 ./
drwxr-x--- 5 nova nova      4096 Jun 25 18:21 ../
-rw-r----- 1 nova nova     31316 Jun 25 17:47 console.log
-rw-r----- 1 root root 189399040 Jun 26 10:38 disk
-rw-r----- 1 nova nova        79 Jun 25 17:46 disk.info

After fixing the permissions back to nova.nova and restarting the unit,
the console.log is now owned by root, and the disk by libvert-qumu.kvm.

root at juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 184924
drwxr-x--- 2 nova         nova      4096 Jun 26 10:39 ./
drwxr-x--- 5 nova         nova      4096 Jun 26 10:38 ../
-rw------- 1 root         root      8039 Jun 26 10:39 console.log
-rw-r----- 1 libvirt-qemu kvm  189399040 Jun 26 10:39 disk
-rw-r----- 1 nova         nova        79 Jun 25 17:46 disk.info

When this unit is stopped, it again goes back to root.root.

root at juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7# ll
total 326324
drwxr-x--- 2 nova nova      4096 Jun 26 10:39 ./
drwxr-x--- 5 nova nova      4096 Jun 26 10:38 ../
-rw------- 1 root root     29040 Jun 26 10:41 console.log
-rw-r----- 1 root root 334168064 Jun 26 10:41 disk
-rw-r----- 1 nova nova        79 Jun 25 17:46 disk.info
root at juju-ed722c-mojo-17:/var/lib/nova/instances/499b7f90-4e5e-4eb9-b29b-2befe835cbe7#

The only real solution, for the moment, is to revert the permissions (as
indicated in the summary) back to world readable for the disk and
../_disk/* files.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to nova in Ubuntu.
https://bugs.launchpad.net/bugs/1885269

Title:
  ussuri: unable to stop and start an instance post upgrade

Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Triaged
Status in nova package in Ubuntu:
  Triaged
Status in nova source package in Focal:
  Triaged
Status in nova source package in Groovy:
  Triaged

Bug description:
  bionic or focal with OpenStack Ussuri.

  The latest version of the nova package sets permissions on package
  update for /var/lib/nova:

    find /var/lib/nova -exec chown nova:nova "{}" +
    find /var/lib/nova -type f -exec chmod 0640 "{}" + -o -type d -exec chmod 0750 "{}" +

  However, when managing vm's via libvirt, various ownership changes
  happen to the underlying disks that mean that nova can no longer
  access the disk files with 0640 permissions.

  The disks (and base image) for a vm are created as nova:nova; libvirt
  then shifts the ownership to libvirt-qemu:kvm as the vm starts.  When
  the vm is stopped the ownership reverts to root:root.

  0640 permissions are maintained - however nova is not part of the root
  or kvm groups so cannot access the file - which means the instance
  cannot be restarted.

  The following permissions are required for correct operation:

    find /var/lib/nova -type f -exec chmod 0644 "{}" + -o -type d -exec
  chmod 0755 "{}" +

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1885269/+subscriptions