[Bug 1697729] Re: port allocator allocates the same SPICE port for multiple guests (race condition)

[Alexander J. Maidak]

Hi,

I recreated the issue in our environment with: 2.5.0-3ubuntu5~cloud0.
The first time I attempted to provision 10 instances at the same time
many errorred out with the same error described in the bug report (qemu-
system-x86_64: failed to initialize spice server)

I upgraded to: 2.5.0-3ubuntu9~cloud0 from pike-proposed.

I attempted to recreate the issue by provisioning 10 instances at once,
I did this twice.  All instances provisioned successfully without error.
I believe 2.5.0-3ubuntu9~cloud0 has resolved this issue for us.  Would
be nice to see it go back to UCA ocata.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1697729

Title:
  port allocator allocates the same SPICE port for multiple guests (race
  condition)

Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive ocata series:
  Triaged
Status in Ubuntu Cloud Archive pike series:
  Fix Committed
Status in libvirt package in Ubuntu:
  Fix Released
Status in libvirt source package in Zesty:
  Fix Committed
Status in libvirt source package in Artful:
  Fix Released

Bug description:
  [Impact]

   * VMs start to fail depending on a race around spice port allocation

   * Solution is the Backport of an upstream fix that avoids a double
     release on the ports

  [Test Case]

   * Prepare a set of VMs using spice and start them concurrently.
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
  $ sed 's/vnc/spice/' /usr/share/uvtool/libvirt/template.xml > spice-template.xml
  $ for idx in {1..20}; do uvt-kvm create --template spice-template.xml --password=ubuntu test-${idx} release=xenial arch=amd64 label=daily; done
  $ for idx in {1..20}; do virsh shutdown test-${idx}; done
  # wait until all are gone
  $ for idx in {1..20}; do (virsh start test-${idx} &); done
  $ for idx in {1..20}; do virsh domdisplay test-${idx} ; done | sort

  * expectation - all work, ports are used one by one
  * current status - failing to intialize:
    error: internal error: process exited while connecting to monitor: ((null):31733): Spice-Warning **: reds.c:2493:reds_init_socket: reds_init_socket: binding socket to 127.0.0.1:5901 failed

  
  [Regression Potential]

   * It is race after all, so we might miss some corner cases in the
     testing, but reviewing the patch and given the verifications so far it
     should be safe. From the patch the change is like:
       Old: Spice-Init -> Cleanup -> Release [...] QemuStop -> Release
                                               ^
                  If new alloc in this time it was released unintentionally
       New: Spice-Init -> Fail            [...] QemuStop -> Release
     This eliminates the race, but still releases the port as intended.

   * This change only affects users of spice ports.

  [Other Info]

   * n/a

  ---

  Using the UCA ocata release of libvirt we sporatically recieve this
  error message in nova-compute.log:

  2017-06-12 14:32:54.359 19007 ERROR nova.compute.manager [instance:
  d1af2a13-0a53-4d9c-ada3-683e4973f28a] libvirtError: internal error:
  process exited while connecting to monitor: ((null):63256): Spice-
  Warning **: reds.c:2463:reds_init_socket: reds_init_socket: binding
  socket to 10.141.112.21:5900 failed

  Please backport the fix for the following bug into UCA ocata/pike releases:
  https://bugzilla.redhat.com/show_bug.cgi?id=1397440

  The patch is documented here:
  https://www.spinics.net/linux/fedora/libvir/msg144093.html

  We've tested backporting this same fix using the ocata UCA libvirt
  2.5.0-3ubuntu5~cloud0 source package and it fixes the problem for us.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1697729/+subscriptions