[Bug 1803137] Re: Unattended-upgrades may keep running after unmounting local filesystems in InstallOnShutdown mode
Launchpad Bug Tracker
1803137 at bugs.launchpad.net
Mon Dec 3 15:13:09 UTC 2018
This bug was fixed in the package unattended-upgrades -
1.5ubuntu3.18.10.0
---------------
unattended-upgrades (1.5ubuntu3.18.10.0) cosmic; urgency=medium
* Revert to running unattended-upgrades.service in multi-user.target
* Trigger unattended-upgrade-shutdown actions with PrepareForShutdown()
Performing upgrades in service's ExecStop did not work when the upgrades
involved restarting services because systemd blocked other stop/start
actions making maintainer scripts time out and be killed leaving a broken
system behind.
Running unattended-upgrades.service before shutdown.target as a oneshot
service made it run after unmounting filesystems and scheduling services
properly on shutdown is a complex problem and adding more services to the
mix make it even more fragile.
The solution of monitoring PrepareForShutdown() signal from DBus
allows Unattended Upgrade to run _before_ the jobs related to shutdown are
queued thus package upgrades can safely restart services without
risking causing deadlocks or breaking part of the shutdown actions.
Also ask running unattended-upgrades to stop when shutdown starts even in
InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to
UnattendedUpgradesShutdown class. (LP: #1778219, LP: #1803137)
* Handle reverting to WantedBy=multi-user.target
* Increase logind's InhibitDelayMaxSec to 30s.
This allows more time for unattended-upgrades to shut down gracefully
or even install a few packages in InstallOnShutdown mode, but is still a
big step back from the 30 minutes allowed for InstallOnShutdown previously.
Users enabling InstallOnShutdown mode are advised to increase
InhibitDelayMaxSec even further possibly to 30 minutes.
* Cache polling result for PreparingForShutdown after it becomes true
* debian/tests/test-systemd.py: Reboot system with dbus call to honor
inhibitor locks
* Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown
changes
* Stop using ActionGroups, they interfere with apt.Cache.clear()
causing all autoremovable packages to be handled as newly autoremovable ones
and be removed by default. Dropping ActionGroup usage does not slow down the
most frequent case of not having anything to upgrade and when ther are
packages to upgrade the gain is small compared to the actual package
installation.
Also collect autoremovable packages before adjusting candidates because that
also changed .is_auto_removable attribute of some of them. (LP: #1803749)
(Closes: #910874)
-- Balint Reczey <rbalint at ubuntu.com> Mon, 26 Nov 2018 12:28:55 +0100
** Changed in: unattended-upgrades (Ubuntu Cosmic)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1803137
Title:
Unattended-upgrades may keep running after unmounting local
filesystems in InstallOnShutdown mode
Status in unattended-upgrades package in Ubuntu:
Fix Released
Status in unattended-upgrades source package in Cosmic:
Fix Released
Bug description:
[Impact]
* Unattended-upgrades may keep running during shutdown even beyond unmounting local filesystems potentially leaving a broken installation behind.
* The fix reverts the original fix of LP: #1778219 and applies a different one starting unattended-upgrades-shutdown _before_ the shutdown transaction starts/
[Test Case]
* Run unattended-upgrades in InstallOnShutdown mode and observe it
being finished installing a few packages _before_ the shutdown
transaction starts:
$ lxc launch ubuntu:18.10 cc-uu-onshutdown
Creating cc-uu-onshutdown
Starting cc-uu-onshutdown
$ lxc shell cc-uu-onshutdown
mesg: ttyname failed: No such device
root at cc-uu-onshutdown:~# apt update -qq
33 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at cc-uu-onshutdown:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
root at cc-uu-onshutdown:~# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
method return time=1542112922.046290 sender=:1.4 -> destination=:1.13 serial=27 reply_serial=2
Session terminated, terminating shell...$
$ lxc shell cc-uu-onshutdown
mesg: ttyname failed: No such device
root at cc-uu-onshutdown:~# journalctl -l
...
Nov 13 12:50:10 cc-uu-shutdown systemd[1]: Started Unattended Upgrades Shutdown.
Nov 13 12:51:17 cc-uu-shutdown systemd-logind[228]: System is rebooting.
...
root at cc-uu-shutdown:~# cat /var/log/unattended-upgrades/unattended-upgrades-shutdown.log
...
2018-11-13 12:51:13,835 WARNING - Running unattended-upgrades in shutdown mode
2018-11-13 12:51:13,852 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
2018-11-13 12:51:15,482 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
2018-11-13 12:51:17,151 WARNING - Unattended-upgrade in progress during shutdown, please don't turn off the computer
2018-11-13 12:51:17,166 INFO - All upgrades installed
[Regression Potential]
* The change reverts the behavior of unattended-upgrades-shutdown to
be close to the behavior observed in Ubuntu 18.04, blocking the
shutdown process reliably until is unattended-upgrades finished, but
it starts _before_ shutdown triggered by the PrepareForShutdown()
signal. Due to the shutdown not fully starting yet users may not get
visual notification of unattended-upgrades running.
In my testing on Bionic when a logged-in user shuts down the system
when InstallOnShutdown is configured the user is dropped out to the
login manager and nothing shows that u-u is running behind the screens
for 30s, when the inhibition timer expires and u-u is starting to
gracefully stop and the usual text appears after the login manager
exits and plymouth shutdown screen is shown. The fix would be the
login manager also monitoring PrepareForShutdown() and exiting (LP:
#1803581).
On Xenial (with Unity) starting shutdown from the graphical session
does not log the user out nor show any progress on the shutdown until
the inhibition timer expires which is confusing. Users should be
logged out on PrepareForShutdown() (LP: #1803581).
* The reversion of unattended-upgrades.service was not complete in
u-u 1.7 and needed a further fix in 1.7ubuntu1 (which fix is already
included in 1.5ubuntu4). This part may be source of potential
regressions.
* The fix itself rewrites big part of unattended-upgrades-shutdown
and this rewrite could cause regressions in running unattended-
upgrades in InstallOnShutdown mode, but also this mode was extensively
tested.
* The fix includes increasing logind's InhibitDelayMaxSec to 30s to
give u-u enough time to gracefully stop in normal mode and install
packages in InstallOnShutdown mode. The delay is global, thus any
other program holding the lock can delay shutdown or sleep by 30s
instead of the original 5s default. This regression is hard to avoid
and the 30s was chosen to minimize the regression potential while
still giving more than 5s to u-u to finish actions. Bugs reporting
increased delay to sleep or shutdown should be monitored to catch
other programs misbehaving with this new default.
[Other Info]
This is a regression of introduced in LP: #1778219 and can be observed
in a cosmic lxd container easily:
$ lxc launch ubuntu:18.10 cc-uu-onshutdown
Creating cc-uu-onshutdown
Starting cc-uu-onshutdown
$ lxc shell cc-uu-onshutdown
mesg: ttyname failed: No such device
root at cc-uu-onshutdown:~# apt update -qq
33 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at cc-uu-onshutdown:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
root at cc-uu-onshutdown:~# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
method return time=1542112922.046290 sender=:1.4 -> destination=:1.13 serial=27 reply_serial=2
Session terminated, terminating shell...$
$ lxc shell cc-uu-onshutdown
mesg: ttyname failed: No such device
root at cc-uu-onshutdown:~# journalctl -l
...
Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: Stopped target Local File Systems.
Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: unattended-upgrades.service: Failed to reset devices.list: Operation not permitted
Nov 13 12:42:02 cc-uu-onshutdown systemd[1]: Starting Unattended Upgrades Shutdown...
..
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1803137/+subscriptions
More information about the foundations-bugs
mailing list