[Bug 1803137] Re: Unattended-upgrades may keep running after unmounting local filesystems in InstallOnShutdown mode

Tue Nov 13 16:31:42 UTC 2018

** Description changed:

  [Impact]

-  * Unattended-upgrades may keep running during shutdown even beyond unmounting local filesystems potentially leaving a broken installation behind.
-  * The fix reverts the original fix of LP: #1778219 and applies a different one startin unattended-upgrades-shutdown _before_ the shutdown transaction starts/
+  * Unattended-upgrades may keep running during shutdown even beyond unmounting local filesystems potentially leaving a broken installation behind.
+  * The fix reverts the original fix of LP: #1778219 and applies a different one startin unattended-upgrades-shutdown _before_ the shutdown transaction starts/

  [Test Case]

-  * Run unattended-upgrades in InstallOnShutdown mode and observe it
+  * Run unattended-upgrades in InstallOnShutdown mode and observe it
  being finished installing a few packages _before_ the shutdown
  transaction starts:

  $ lxc launch ubuntu:18.10 cc-uu-onshutdown
  Creating cc-uu-onshutdown
  Starting cc-uu-onshutdown
- $ lxc shell cc-uu-onshutdown 
+ $ lxc shell cc-uu-onshutdown
  mesg: ttyname failed: No such device
  root at cc-uu-onshutdown:~# apt update -qq
  33 packages can be upgraded. Run 'apt list --upgradable' to see them.
  root at cc-uu-onshutdown:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
  root at cc-uu-onshutdown:~# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
  method return time=1542112922.046290 sender=:1.4 -> destination=:1.13 serial=27 reply_serial=2

  Session terminated, terminating shell...$
- $ lxc shell cc-uu-onshutdown 
+ $ lxc shell cc-uu-onshutdown
  mesg: ttyname failed: No such device
  root at cc-uu-onshutdown:~# journalctl -l
  ...
  Nov 13 12:50:10 cc-uu-shutdown systemd[1]: Started Unattended Upgrades Shutdown.
  Nov 13 12:51:17 cc-uu-shutdown systemd-logind[228]: System is rebooting.

  ...
  root at cc-uu-shutdown:~# cat /var/log/unattended-upgrades/unattended-upgrades-shutdown.log
  ...
  2018-11-13 12:51:13,835 WARNING - Running unattended-upgrades in shutdown mode
  2018-11-13 12:51:13,852 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
  2018-11-13 12:51:15,482 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
  2018-11-13 12:51:17,151 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
  2018-11-13 12:51:17,166 INFO - All upgrades installed

+ [Regression Potential]

- [Regression Potential] 
+  * The change reverts the behavior of unattended-upgrades-shutdown to be
+ close to the behavior observed in Ubuntu 18.04, blocking the shutdown
+ process reliably until is unattended-upgrades finished, but it starts
+ _before_ shutdown triggered by the PrepareForShutdown() signal. Due to
+ the shutdown not fully starting yet users may not get visual
+ notification of unattended-upgrades running. In my testing when a
+ logged-in user shuts down the system when InstallOnShutdown is
+ configured the user is dropped out to the login manager and nothing
+ shows that u-u is running behind the screens for 30s, when the
+ inhibition timer expires and u-u is starting to gracefully stop and the
+ usual text appears after the login manager exits and plymouth shutdown
+ screen is shown. The fix would be the login manager also monitoring
+ PrepareForShutdown() and exiting.

-  * The change reverts the behavior of unattended-upgrades-shutdown to be close to the behavior observed in Ubuntu 18.04, blocking the shutdown process reliably until is unattended-upgrades finished, but it starts _before_ shutdown triggered by the PrepareForShutdown() signal. Due to the shutdown not fully starting yet users may not get visual notification of unattended-upgrades running in some system configuration (testing that is WIP).
-  * The reversion of unattended-upgrades.service was not complete in u-u 1.7 and needed a further fix in 1.7ubuntu1 (which fix is already included in 1.5ubuntu4). This part may be source of potential regressions.
-  * The fix itself rewrites big part of unattended-upgrades-shutdown and this rewrite could cause regressions in running unattended-upgrades in InstallOnShutdown mode, but also this mode was extensively tested.
-  * The fix includes increasing logind's InhibitDelayMaxSec to 30s to give u-u enough time to gracefully stop in normal mode and install packages in InstallOnShutdown mode. The delay is global, thus any other program holding the lock can delay shutdown or sleep by 30s instead of the original 5s default. This regression is hard to avoid and the 30s was chosen to minimize the regression potential while still giving more than 5s to u-u to finish actions. Bugs reporting increased delay to sleep or shutdown should be monitored to catch other programs misbehaving with this new default.
+  * The reversion of unattended-upgrades.service was not complete in u-u
+ 1.7 and needed a further fix in 1.7ubuntu1 (which fix is already
+ included in 1.5ubuntu4). This part may be source of potential
+ regressions.

+  * The fix itself rewrites big part of unattended-upgrades-shutdown and
+ this rewrite could cause regressions in running unattended-upgrades in
+ InstallOnShutdown mode, but also this mode was extensively tested.
+ 
+  * The fix includes increasing logind's InhibitDelayMaxSec to 30s to
+ give u-u enough time to gracefully stop in normal mode and install
+ packages in InstallOnShutdown mode. The delay is global, thus any other
+ program holding the lock can delay shutdown or sleep by 30s instead of
+ the original 5s default. This regression is hard to avoid and the 30s
+ was chosen to minimize the regression potential while still giving more
+ than 5s to u-u to finish actions. Bugs reporting increased delay to
+ sleep or shutdown should be monitored to catch other programs
+ misbehaving with this new default.

  [Other Info]
-  
- This is a regression of introduced in LP: #1778219 and can be observed in a cosmic lxd container easily:
+ 
+ This is a regression of introduced in LP: #1778219 and can be observed
+ in a cosmic lxd container easily:

  $ lxc launch ubuntu:18.10 cc-uu-onshutdown
  Creating cc-uu-onshutdown
  Starting cc-uu-onshutdown
- $ lxc shell cc-uu-onshutdown 
+ $ lxc shell cc-uu-onshutdown
  mesg: ttyname failed: No such device
  root at cc-uu-onshutdown:~# apt update -qq
  33 packages can be upgraded. Run 'apt list --upgradable' to see them.
  root at cc-uu-onshutdown:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
  root at cc-uu-onshutdown:~# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
  method return time=1542112922.046290 sender=:1.4 -> destination=:1.13 serial=27 reply_serial=2

  Session terminated, terminating shell...$
- $ lxc shell cc-uu-onshutdown 
+ $ lxc shell cc-uu-onshutdown
  mesg: ttyname failed: No such device
  root at cc-uu-onshutdown:~# journalctl -l
  ...
  Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: Stopped target Local File Systems.
  Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: unattended-upgrades.service: Failed to reset devices.list: Operation not permitted
  Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: Starting Unattended Upgrades Shutdown...
  ..

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1803137

Title:
  Unattended-upgrades may keep running after unmounting local
  filesystems in InstallOnShutdown mode

Status in unattended-upgrades package in Ubuntu:
  Fix Released

Bug description:
  [Impact]

   * Unattended-upgrades may keep running during shutdown even beyond unmounting local filesystems potentially leaving a broken installation behind.
   * The fix reverts the original fix of LP: #1778219 and applies a different one startin unattended-upgrades-shutdown _before_ the shutdown transaction starts/

  [Test Case]

   * Run unattended-upgrades in InstallOnShutdown mode and observe it
  being finished installing a few packages _before_ the shutdown
  transaction starts:

  $ lxc launch ubuntu:18.10 cc-uu-onshutdown
  Creating cc-uu-onshutdown
  Starting cc-uu-onshutdown
  $ lxc shell cc-uu-onshutdown
  mesg: ttyname failed: No such device
  root at cc-uu-onshutdown:~# apt update -qq
  33 packages can be upgraded. Run 'apt list --upgradable' to see them.
  root at cc-uu-onshutdown:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
  root at cc-uu-onshutdown:~# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
  method return time=1542112922.046290 sender=:1.4 -> destination=:1.13 serial=27 reply_serial=2

  Session terminated, terminating shell...$
  $ lxc shell cc-uu-onshutdown
  mesg: ttyname failed: No such device
  root at cc-uu-onshutdown:~# journalctl -l
  ...
  Nov 13 12:50:10 cc-uu-shutdown systemd[1]: Started Unattended Upgrades Shutdown.
  Nov 13 12:51:17 cc-uu-shutdown systemd-logind[228]: System is rebooting.

  ...
  root at cc-uu-shutdown:~# cat /var/log/unattended-upgrades/unattended-upgrades-shutdown.log
  ...
  2018-11-13 12:51:13,835 WARNING - Running unattended-upgrades in shutdown mode
  2018-11-13 12:51:13,852 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
  2018-11-13 12:51:15,482 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
  2018-11-13 12:51:17,151 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
  2018-11-13 12:51:17,166 INFO - All upgrades installed

  [Regression Potential]

   * The change reverts the behavior of unattended-upgrades-shutdown to
  be close to the behavior observed in Ubuntu 18.04, blocking the
  shutdown process reliably until is unattended-upgrades finished, but
  it starts _before_ shutdown triggered by the PrepareForShutdown()
  signal. Due to the shutdown not fully starting yet users may not get
  visual notification of unattended-upgrades running. In my testing when
  a logged-in user shuts down the system when InstallOnShutdown is
  configured the user is dropped out to the login manager and nothing
  shows that u-u is running behind the screens for 30s, when the
  inhibition timer expires and u-u is starting to gracefully stop and
  the usual text appears after the login manager exits and plymouth
  shutdown screen is shown. The fix would be the login manager also
  monitoring PrepareForShutdown() and exiting.

   * The reversion of unattended-upgrades.service was not complete in
  u-u 1.7 and needed a further fix in 1.7ubuntu1 (which fix is already
  included in 1.5ubuntu4). This part may be source of potential
  regressions.

   * The fix itself rewrites big part of unattended-upgrades-shutdown
  and this rewrite could cause regressions in running unattended-
  upgrades in InstallOnShutdown mode, but also this mode was extensively
  tested.

   * The fix includes increasing logind's InhibitDelayMaxSec to 30s to
  give u-u enough time to gracefully stop in normal mode and install
  packages in InstallOnShutdown mode. The delay is global, thus any
  other program holding the lock can delay shutdown or sleep by 30s
  instead of the original 5s default. This regression is hard to avoid
  and the 30s was chosen to minimize the regression potential while
  still giving more than 5s to u-u to finish actions. Bugs reporting
  increased delay to sleep or shutdown should be monitored to catch
  other programs misbehaving with this new default.

  [Other Info]

  This is a regression of introduced in LP: #1778219 and can be observed
  in a cosmic lxd container easily:

  $ lxc launch ubuntu:18.10 cc-uu-onshutdown
  Creating cc-uu-onshutdown
  Starting cc-uu-onshutdown
  $ lxc shell cc-uu-onshutdown
  mesg: ttyname failed: No such device
  root at cc-uu-onshutdown:~# apt update -qq
  33 packages can be upgraded. Run 'apt list --upgradable' to see them.
  root at cc-uu-onshutdown:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
  root at cc-uu-onshutdown:~# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
  method return time=1542112922.046290 sender=:1.4 -> destination=:1.13 serial=27 reply_serial=2

  Session terminated, terminating shell...$
  $ lxc shell cc-uu-onshutdown
  mesg: ttyname failed: No such device
  root at cc-uu-onshutdown:~# journalctl -l
  ...
  Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: Stopped target Local File Systems.
  Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: unattended-upgrades.service: Failed to reset devices.list: Operation not permitted
  Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: Starting Unattended Upgrades Shutdown...
  ..

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1803137/+subscriptions