[Bug 1681761] Re: instance destroy needs better guards for zfs ephemeral storage volumes

[Chris MacNaughton]

** Changed in: nova-lxd (Ubuntu)
       Status: Triaged => Won't Fix

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1681761

Title:
  instance destroy needs better guards for zfs ephemeral storage volumes

Status in Ubuntu Cloud Archive:
  Won't Fix
Status in nova-lxd:
  Won't Fix
Status in nova-lxd package in Ubuntu:
  Won't Fix

Bug description:
  If instance creation fails for any reason, then its tricky to delete
  instances on a ZFS storage backend with ephemeral storage volumes; the
  destroy call does not check whether the volume has actually been
  created, resulting in a stacktrace and failure to delete the volume:

  2017-04-11 07:58:07.400 26969 DEBUG oslo_concurrency.processutils [req-b18af233-416b-4567-8b33-6875a6800bcb 9a68aab1a4764787acd6df2fc3cb8dcd 96c08cea38fb4daeb74f021e17f3c04f - - -] u'sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy lxd/instance-0000086b-ephemeral' failed. Not Retrying. execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:433
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [req-b18af233-416b-4567-8b33-6875a6800bcb 9a68aab1a4764787acd6df2fc3cb8dcd 96c08cea38fb4daeb74f021e17f3c04f - - -] [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Instance failed to spawn
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Traceback (most recent call last):
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2125, in _build_resources
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     yield resources
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     block_device_info=block_device_info)
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 392, in spawn
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     context, instance, network_info, block_device_info)
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 482, in destroy
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     context, instance, network_info, block_device_info)
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 496, in cleanup
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     storage.detach_ephemeral(block_device_info, lxd_config, instance)
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/storage.py", line 109, in detach_ephemeral
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     run_as_root=True)
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 297, in execute
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 180, in execute
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     return processutils.execute(*cmd, **kwargs)
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 400, in execute
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]     cmd=sanitized_cmd)
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] ProcessExecutionError: Unexpected error while running command.
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy lxd/instance-0000086b-ephemeral
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Exit code: 1
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Stdout: u''
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Stderr: u"cannot open 'lxd/instance-0000086b-ephemeral': dataset does not exist\n"
  2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: python-nova-lxd 15.0.0-0ubuntu1~cloud0 [modified: usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py usr/lib/python2.7/dist-packages/nova/virt/lxd/storage.py] [origin: Canonical]
  ProcVersionSignature: Ubuntu 4.4.0-72.93-generic 4.4.49
  Uname: Linux 4.4.0-72-generic x86_64
  NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
  ApportVersion: 2.20.1-0ubuntu2.5
  Architecture: amd64
  CrashDB:
   {
                  "impl": "launchpad",
                  "project": "cloud-archive",
                  "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
               }
  Date: Tue Apr 11 10:18:33 2017
  PackageArchitecture: all
  ProcEnviron:
   TERM=screen-256color-bce
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: nova-lxd
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1681761/+subscriptions