[Bug 1654600] Re: unattended-upgrade-shutdown hangs when /var is a separate filesystem

[Louis Bouchard]

Here is a recap of my day of work :

The unattended-upgrades.service unit never runs the ExecStop because it
is not enabled as we see here :

# systemctl status unattended-upgrades.service 
● unattended-upgrades.service - Unattended Upgrades Shutdown
   Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
   Active: inactive (dead)           <<<<<<<<<<<<<<<
     Docs: man:unattended-upgrade(8)

This is caused by a missing "Default-Start" header in /etc/init.d
/unattended-upgrades. Trying to enable the unit leads to :

# systemctl enable unattended-upgrades.service
Synchronizing state of unattended-upgrades.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable unattended-upgrades
update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, aborting.

I'm currently working on fixing all this in the packaging so we get a
correctly working unit after the upgrade of the package.

I have also reworked the unit :

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
After=network.target local-fs.target
RequiresMountsFor=/var/log /var/run
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStopSec=900

[Install]
WantedBy=multi-user.target

With this configuration, the unit runs the ExecStop as expected. I also
tested using Unattended-Upgrade::InstallOnShutdown "true" and it works
as expected, which is to block the shutdown for upgrade with the network
being available.

I should have that available for testing tomorrow.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1654600

Title:
  unattended-upgrade-shutdown hangs when /var is a separate filesystem

Status in unattended-upgrades package in Ubuntu:
  In Progress
Status in unattended-upgrades source package in Xenial:
  Triaged
Status in unattended-upgrades source package in Yakkety:
  Triaged
Status in unattended-upgrades package in Debian:
  New

Bug description:
  [SRU justification]
  This fix is needed to make sure that the system does not hang on shutdown when /var is a speparate file system

  [Impact]
  System can hang up to 10 minutes if not fixed.

  [Fix]
  Change the systemd unit's ExecStart to an ExecStop so the unit is correctly sequenced.

  [Test Case]
  In a VM with /var separated from the / file system, shutdown the VM. It will hang for 10 minutes

  [Regression]
  None to be expected. A ExecStop unit will be sequenced before the Before= units which is earlier than previously.

  [Original description of the problem]
  The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
  process during shutdown. This unit file is running together with all filesystem
  unmount services.

  The unattended-upgrades service checks if the lockfile for unattended-upgrade
  (in /var/run) exists, and if it does, there is an unattended-upgrade in progress
  and the service will wait until it finishes (and therefore automatically wait at
  shutdown).

  However, if /var is a separate filesystem, it will get unmounted even though /var/run
  is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem.
  The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and
  tries to check the lockfile again. After 10 minutes (the default timeout), it will finally
  exit and the system will continue shutdown.

  The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown
  where it tries to lock itself:

      while True:
          res = apt_pkg.get_lock(options.lock_file)
          logging.debug("get_lock returned %i" % res)
          # exit here if there is no lock
          if res > 0:
              logging.debug("lock not taken")
              break
          lock_was_taken = True

  The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error.
  File descriptors are just C file descriptors, so they are always positive integers.
  The code should check the result to be negative, not positive. I have attached a patch
  to reverse the logic.

  Additional information:

  1)
  Description:	Ubuntu 16.04.1 LTS
  Release:	16.04

  2)
  unattended-upgrades:
    Installed: 0.90ubuntu0.3
    Candidate: 0.90ubuntu0.3
    Version table:
   *** 0.90ubuntu0.3 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
          100 /var/lib/dpkg/status
       0.90 500
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
          500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
  3)
  Fast reboot
  4)
  Very slow reboot (after a 10 minutes timeout)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1654600/+subscriptions