[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem

[Louis Bouchard]

@kay-diam:

I get what you mean. I also use KVM to debug the issue.

I'm going to switch to verification-failed to this one does not get
released and push the investigation further as there is another issue
that side-tracked me :

If Unattended-Upgrade::InstallOnShutdown is set to "true", it will still
fail as the network will no longer be available when it gets to access
the archive so this must also be fixed.

...Louis

** Tags removed: verification-needed
** Tags added: verification-failed

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1654600

Title:
  unattended-upgrade-shutdown hangs when /var is a separate filesystem

Status in unattended-upgrades package in Ubuntu:
  Fix Released
Status in unattended-upgrades source package in Xenial:
  Fix Committed
Status in unattended-upgrades source package in Yakkety:
  Fix Committed
Status in unattended-upgrades package in Debian:
  New

Bug description:
  [SRU justification]
  This fix is needed to make sure that the system does not hang on shutdown when /var is a speparate file system

  [Impact]
  System can hang up to 10 minutes if not fixed.

  [Fix]
  Change the systemd unit's ExecStart to an ExecStop so the unit is correctly sequenced.

  [Test Case]
  In a VM with /var separated from the / file system, shutdown the VM. It will hang for 10 minutes

  [Regression]
  None to be expected. A ExecStop unit will be sequenced before the Before= units which is earlier than previously.

  [Original description of the problem]
  The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
  process during shutdown. This unit file is running together with all filesystem
  unmount services.

  The unattended-upgrades service checks if the lockfile for unattended-upgrade
  (in /var/run) exists, and if it does, there is an unattended-upgrade in progress
  and the service will wait until it finishes (and therefore automatically wait at
  shutdown).

  However, if /var is a separate filesystem, it will get unmounted even though /var/run
  is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem.
  The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and
  tries to check the lockfile again. After 10 minutes (the default timeout), it will finally
  exit and the system will continue shutdown.

  The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown
  where it tries to lock itself:

      while True:
          res = apt_pkg.get_lock(options.lock_file)
          logging.debug("get_lock returned %i" % res)
          # exit here if there is no lock
          if res > 0:
              logging.debug("lock not taken")
              break
          lock_was_taken = True

  The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error.
  File descriptors are just C file descriptors, so they are always positive integers.
  The code should check the result to be negative, not positive. I have attached a patch
  to reverse the logic.

  Additional information:

  1)
  Description:	Ubuntu 16.04.1 LTS
  Release:	16.04

  2)
  unattended-upgrades:
    Installed: 0.90ubuntu0.3
    Candidate: 0.90ubuntu0.3
    Version table:
   *** 0.90ubuntu0.3 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
          100 /var/lib/dpkg/status
       0.90 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
  3)
  Fast reboot
  4)
  Very slow reboot (after a 10 minutes timeout)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions