[Bug 1803749] Re: apt.Cache.clear() may set apt.Package.is_auto_removable to False
Launchpad Bug Tracker
1803749 at bugs.launchpad.net
Mon Dec 3 15:13:09 UTC 2018
This bug was fixed in the package unattended-upgrades -
1.5ubuntu3.18.10.0
---------------
unattended-upgrades (1.5ubuntu3.18.10.0) cosmic; urgency=medium
* Revert to running unattended-upgrades.service in multi-user.target
* Trigger unattended-upgrade-shutdown actions with PrepareForShutdown()
Performing upgrades in service's ExecStop did not work when the upgrades
involved restarting services because systemd blocked other stop/start
actions making maintainer scripts time out and be killed leaving a broken
system behind.
Running unattended-upgrades.service before shutdown.target as a oneshot
service made it run after unmounting filesystems and scheduling services
properly on shutdown is a complex problem and adding more services to the
mix make it even more fragile.
The solution of monitoring PrepareForShutdown() signal from DBus
allows Unattended Upgrade to run _before_ the jobs related to shutdown are
queued thus package upgrades can safely restart services without
risking causing deadlocks or breaking part of the shutdown actions.
Also ask running unattended-upgrades to stop when shutdown starts even in
InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to
UnattendedUpgradesShutdown class. (LP: #1778219, LP: #1803137)
* Handle reverting to WantedBy=multi-user.target
* Increase logind's InhibitDelayMaxSec to 30s.
This allows more time for unattended-upgrades to shut down gracefully
or even install a few packages in InstallOnShutdown mode, but is still a
big step back from the 30 minutes allowed for InstallOnShutdown previously.
Users enabling InstallOnShutdown mode are advised to increase
InhibitDelayMaxSec even further possibly to 30 minutes.
* Cache polling result for PreparingForShutdown after it becomes true
* debian/tests/test-systemd.py: Reboot system with dbus call to honor
inhibitor locks
* Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown
changes
* Stop using ActionGroups, they interfere with apt.Cache.clear()
causing all autoremovable packages to be handled as newly autoremovable ones
and be removed by default. Dropping ActionGroup usage does not slow down the
most frequent case of not having anything to upgrade and when ther are
packages to upgrade the gain is small compared to the actual package
installation.
Also collect autoremovable packages before adjusting candidates because that
also changed .is_auto_removable attribute of some of them. (LP: #1803749)
(Closes: #910874)
-- Balint Reczey <rbalint at ubuntu.com> Mon, 26 Nov 2018 12:28:55 +0100
** Changed in: unattended-upgrades (Ubuntu Cosmic)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1803749
Title:
apt.Cache.clear() may set apt.Package.is_auto_removable to False
Status in unattended-upgrades package in Ubuntu:
Fix Released
Status in unattended-upgrades source package in Bionic:
Fix Released
Status in unattended-upgrades source package in Cosmic:
Fix Released
Bug description:
[Impact]
* Originally autoremovable packages can be removed as newly autoremovable ones by unattended-upgrades
* This can surprise users potentially removing packages which are needed for the system's operation.
[Test Case]
* The buggy u-u version will remove the previously autoremovable zsh
at the end, the fixed one does not.
$ sudo debootstrap bionic uu-test-bionic-1803749
...
$ sudo chroot uu-test-bionic-1803749
# apt install unattended-upgrades zsh
# apt-mark auto zsh
...
# echo "Unattended-Upgrade::Package-Blacklist {"libs";}" > /etc/apt/apt.conf.d/51unattended-upgrades
# echo "deb http://archive.ubuntu.com/ubuntu bionic-security main" >> /etc/apt/sources.list
# echo "deb http://archive.ubuntu.com/ubuntu bionic-updates main" >> /etc/apt/sources.list
# apt update -qq
# unattended-upgrades --dry-run --verbose --debug
...
All upgrades installed
marking zsh for removal
marking zsh-common for removal
Packages that were successfully auto-removed:
Packages that are kept back:
InstCount=0 DelCount=0 BrokenCount=0
#
[Regression Potential]
* Unattended-upgrades may use more CPU-time for operation but I did not observe a significant increase. Autopkgtest measures u-u's performance thus if this regression occurs, we can observe it easily.
* Due to the code changes u-u may still remove already autoremovable packages or fail to remove newly autoremovable ones in default configuration, but since the code became simpler with the change by eliminating an optimization this regression is unlikely to take place.
[Original Bug Text]
While triaging LP: #1803587 I found that originally autoremovable
packages were handled as newly autoremovable ones by unattended-
upgrades due to not finding all of them at the beginning of u-u's run.
The root cause seems to be cache.clear() resetting
pkg.is_auto_removable to False under some circumstances I haven't
fully narrowed down.
Set up a Bionic (or later) system with autoremovable packages,
packages upgradable from -security and blacklist at least one
upgradable package and apply the following patch to u-u:
root at bb-1803587:~# diff -Naur /usr/bin/unattended-upgrade.orig /usr/bin/unattended-upgrade
--- /usr/bin/unattended-upgrade.orig 2018-11-16 16:17:58.522583254 +0000
+++ /usr/bin/unattended-upgrade 2018-11-16 16:36:12.226675870 +0000
@@ -948,7 +948,9 @@
def rewind_cache(cache, pkgs_to_upgrade):
# type: (apt.Cache, List[apt.Package]) -> None
""" set the cache back to the state with packages_to_upgrade """
+ print([pkg.name for pkg in cache if pkg.is_auto_removable])
cache.clear()
+ print([pkg.name for pkg in cache if pkg.is_auto_removable])
for pkg2 in pkgs_to_upgrade:
pkg2.mark_install(from_user=not pkg2.is_auto_installed)
if cache.broken_count > 0:
Run u-u to observe cache.clear() resetting the list of autoremovable
packages:
~# /usr/bin/unattended-upgrade --dry-run --verbose --debug
Initial blacklisted packages: systemd
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
Using (^linux-image|^linux-headers|^linux-image-extra|^linux-modules|^linux-modules-extra|^linux-signed-image|^kfreebsd-image|^kfreebsd-headers|^gnumach-image|^.*-modules|^.*-kernel|^linux-backports-modules-.*|^linux-modules-.*|^linux-tools|^linux-cloud-tools) regexp to find kernel packages
Using (^linux-image.*4.15.0-38-generic|^linux-headers.*4.15.0-38-generic|^linux-image-extra.*4.15.0-38-generic|^linux-modules.*4.15.0-38-generic|^linux-modules-extra.*4.15.0-38-generic|^linux-signed-image.*4.15.0-38-generic|^kfreebsd-image.*4.15.0-38-generic|^kfreebsd-headers.*4.15.0-38-generic|^gnumach-image.*4.15.0-38-generic|4.15.0-38-generic.*-modules|4.15.0-38-generic.*-kernel|^linux-backports-modules-.*.*4.15.0-38-generic|^linux-modules-.*.*4.15.0-38-generic|^linux-tools.*4.15.0-38-generic|^linux-cloud-tools.*4.15.0-38-generic) regexp to find running kernel packages
Checking: apport ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>])
adjusting candidate version: apport=2.20.9-0ubuntu7.1
...
Checking: libnss-systemd ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>, <Origin component:'main' archive:'bionic-security' origin:'Ubuntu' label:'Ubuntu' site:'security.ubuntu.com' isTrusted:True>])
skipping blacklisted package systemd
pkg systemd package has been blacklisted
sanity check failed
['libfreetype6']
[]
...
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1803749/+subscriptions
More information about the foundations-bugs
mailing list