[Bug 1688508] [NEW] libvirt-guests.sh fails to shutdown guests in parallel

[Launchpad Bug Tracker]

You have been subscribed to a public bug by Dariusz Gadomski (dgadomski):

[Environment]

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial

[Impact]

There is a bug/race condition on libvirt-guests.service, that prevents
the shutdown of guests to happen in parallel.

The critical chain for this service is:

libvirt-guests.service +20ms
└─libvirt-bin.service @2.784s +140ms
  └─remote-fs.target @2.777s
    └─remote-fs-pre.target @2.775s
      └─open-iscsi.service @2.554s +116ms
        └─iscsid.service @2.525s +18ms
          └─network-online.target @2.502s
            └─network.target @1.955s
              └─networking.service @1.625s +299ms
                └─network-pre.target @1.601s
                  └─cloud-init-local.service @405ms +1.072s
                    └─systemd-remount-fs.service @232ms +64ms
                      └─systemd-journald.socket @178ms
                        └─-.slice @117ms

As an example, I have the following kvm host with 42 virtual
machines.

ubuntu at xenial-base:~$ virsh list --all
 Id    Name                           State
----------------------------------------------------
 12    locked-trusty-2                running
 13    locked-trusty-3                running
[...]
 41    locked-trusty-42               running

After rebooting the machine:

[  250.999516] libvirt-guests.sh[4215]: Running guests on default URI: locked-trusty-2, locked-trusty-4, locked-trusty-12, locked-trusty-3, locked-trusty-5, locked-trusty-11, locked-trusty-10, locked-trusty-8, locked-trusty-9, locked-trusty-7, locked-trusty-6, locked-trusty-13, locked-trusty-14, locked-trusty-15, locked-trusty-16, locked-trusty-17, locked-trusty-18, locked-trusty-19, locked-trusty-20, locked-trusty-21, locked-trusty-22, locked-trusty-23, locked-trusty-24, locked-trusty-25, locked-trusty-26, locked-trusty-27, locked-trusty-28, locked-trusty-29, locked-trusty-30, locked-trusty-31, locked-trusty-32, locked-trusty-33, locked-trusty-34, locked-trusty-35, locked-trusty-36, locked-trusty-37, locked-trusty-38, locked-trusty-39, locked-trusty-40, locked-trusty-41, locked-trusty-42
[  251.011367] libvirt-guests.sh[4215]: Shutting down guests on default URI...
[  251.027072] libvirt-guests.sh[4215]: Starting shutdown on guest: locked-trusty-2
[...]
[  391.949941] libvirt-guests.sh[4215]: Waiting for 28 guests to shut down, 10 seconds left
[  398.074405] libvirt-guests.sh[4215]: Waiting for 28 guests to shut down, 5 seconds left
[  403.020479] libvirt-guests.sh[4215]: Timeout expired while shutting down domains
[  OK  ] Stopped Suspend Active Libvirt Guests.
[  OK  ] Stopped target System Time Synchronized.

[Test Case]

 * Make sure the following variables are set in /etc/default/libvirt-
guests (which are all default options):

ON_SHUTDOWN=shutdown
PARALLEL_SHUTDOWN=10
SHUTDOWN_TIMEOUT=120

 * Create over 20 virtual machines (in my case, using uvt-kvm).

$  for f in $(seq 0 40); do uvt-kvm create --memory 2000 --cpu 1 locked-
trusty-$f release=xenial arch=amd64 ; done

 * Reboot the machine and monitor the systemd service stop sequence
or console output.

(With systemd: systemctl start debug-shell and jumpt to ctrl+alt+f9)

* Error message "Timeout expired while shutting down domains" should
be displayed.

[Regression Potential]

* None identified.

[Other Info]

* There is a proposed patch in upstream already that has been already
linked to this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1450141

** Affects: libvirt
     Importance: Undecided
         Status: Fix Released

** Affects: libvirt (Ubuntu)
     Importance: Medium
     Assignee: ChristianEhrhardt (paelzer)
         Status: Fix Released

** Affects: libvirt (Ubuntu Xenial)
     Importance: Medium
     Assignee: Jorge Niedbalski (niedbalski)
         Status: In Progress

** Affects: libvirt (Ubuntu Zesty)
     Importance: Medium
     Assignee: Jorge Niedbalski (niedbalski)
         Status: Won't Fix

** Affects: libvirt (Ubuntu Artful)
     Importance: Medium
     Assignee: Jorge Niedbalski (niedbalski)
         Status: In Progress


** Tags: need-upstream-report patch
-- 
libvirt-guests.sh fails to shutdown guests in parallel
https://bugs.launchpad.net/bugs/1688508
You received this bug notification because you are a member of Ubuntu Sponsors Team, which is subscribed to the bug report.