[Bug 1803749] [NEW] apt.Cache.clear() may set apt.Package.is_auto_removable to False

[Balint Reczey]

Public bug reported:

While triaging LP: #1803587 I found that originally autoremovable
packages were handled as newly autoremovable ones by unattended-upgrades
due to not finding all of them at the beginning of u-u's run.

The root cause seems to be cache.clear() resetting pkg.is_auto_removable
to False under some circumstances I haven't fully narrowed down.

The reproduction is as follows:
Set up a Bionic (or later) system with autoremovable packages, packages upgradable from -security and blacklist at least one upgradable package and apply the following patch to u-u:


root at bb-1803587:~# diff -Naur /usr/bin/unattended-upgrade.orig /usr/bin/unattended-upgrade
--- /usr/bin/unattended-upgrade.orig	2018-11-16 16:17:58.522583254 +0000
+++ /usr/bin/unattended-upgrade	2018-11-16 16:36:12.226675870 +0000
@@ -948,7 +948,9 @@
 def rewind_cache(cache, pkgs_to_upgrade):
     # type: (apt.Cache, List[apt.Package]) -> None
     """ set the cache back to the state with packages_to_upgrade """
+    print([pkg.name  for pkg in cache if pkg.is_auto_removable])
     cache.clear()
+    print([pkg.name for pkg in cache if pkg.is_auto_removable])
     for pkg2 in pkgs_to_upgrade:
         pkg2.mark_install(from_user=not pkg2.is_auto_installed)
     if cache.broken_count > 0:


Run u-u to observe cache.clear() resetting the list of autoremovable packages:

 ~# /usr/bin/unattended-upgrade --dry-run --verbose --debug
Initial blacklisted packages: systemd
Initial whitelisted packages: 
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
Using (^linux-image|^linux-headers|^linux-image-extra|^linux-modules|^linux-modules-extra|^linux-signed-image|^kfreebsd-image|^kfreebsd-headers|^gnumach-image|^.*-modules|^.*-kernel|^linux-backports-modules-.*|^linux-modules-.*|^linux-tools|^linux-cloud-tools) regexp to find kernel packages
Using (^linux-image.*4.15.0-38-generic|^linux-headers.*4.15.0-38-generic|^linux-image-extra.*4.15.0-38-generic|^linux-modules.*4.15.0-38-generic|^linux-modules-extra.*4.15.0-38-generic|^linux-signed-image.*4.15.0-38-generic|^kfreebsd-image.*4.15.0-38-generic|^kfreebsd-headers.*4.15.0-38-generic|^gnumach-image.*4.15.0-38-generic|4.15.0-38-generic.*-modules|4.15.0-38-generic.*-kernel|^linux-backports-modules-.*.*4.15.0-38-generic|^linux-modules-.*.*4.15.0-38-generic|^linux-tools.*4.15.0-38-generic|^linux-cloud-tools.*4.15.0-38-generic) regexp to find running kernel packages
Checking: apport ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>])
adjusting candidate version: apport=2.20.9-0ubuntu7.1
...
Checking: libnss-systemd ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>, <Origin component:'main' archive:'bionic-security' origin:'Ubuntu' label:'Ubuntu' site:'security.ubuntu.com' isTrusted:True>])
skipping blacklisted package systemd
pkg systemd package has been blacklisted
sanity check failed
['libfreetype6']
[]
...

** Affects: python-apt (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python-apt in Ubuntu.
https://bugs.launchpad.net/bugs/1803749

Title:
  apt.Cache.clear() may set apt.Package.is_auto_removable to False

Status in python-apt package in Ubuntu:
  New

Bug description:
  While triaging LP: #1803587 I found that originally autoremovable
  packages were handled as newly autoremovable ones by unattended-
  upgrades due to not finding all of them at the beginning of u-u's run.

  The root cause seems to be cache.clear() resetting
  pkg.is_auto_removable to False under some circumstances I haven't
  fully narrowed down.

  The reproduction is as follows:
  Set up a Bionic (or later) system with autoremovable packages, packages upgradable from -security and blacklist at least one upgradable package and apply the following patch to u-u:

  
  root at bb-1803587:~# diff -Naur /usr/bin/unattended-upgrade.orig /usr/bin/unattended-upgrade
  --- /usr/bin/unattended-upgrade.orig	2018-11-16 16:17:58.522583254 +0000
  +++ /usr/bin/unattended-upgrade	2018-11-16 16:36:12.226675870 +0000
  @@ -948,7 +948,9 @@
   def rewind_cache(cache, pkgs_to_upgrade):
       # type: (apt.Cache, List[apt.Package]) -> None
       """ set the cache back to the state with packages_to_upgrade """
  +    print([pkg.name  for pkg in cache if pkg.is_auto_removable])
       cache.clear()
  +    print([pkg.name for pkg in cache if pkg.is_auto_removable])
       for pkg2 in pkgs_to_upgrade:
           pkg2.mark_install(from_user=not pkg2.is_auto_installed)
       if cache.broken_count > 0:

  
  Run u-u to observe cache.clear() resetting the list of autoremovable packages:

   ~# /usr/bin/unattended-upgrade --dry-run --verbose --debug
  Initial blacklisted packages: systemd
  Initial whitelisted packages: 
  Starting unattended upgrades script
  Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
  Using (^linux-image|^linux-headers|^linux-image-extra|^linux-modules|^linux-modules-extra|^linux-signed-image|^kfreebsd-image|^kfreebsd-headers|^gnumach-image|^.*-modules|^.*-kernel|^linux-backports-modules-.*|^linux-modules-.*|^linux-tools|^linux-cloud-tools) regexp to find kernel packages
  Using (^linux-image.*4.15.0-38-generic|^linux-headers.*4.15.0-38-generic|^linux-image-extra.*4.15.0-38-generic|^linux-modules.*4.15.0-38-generic|^linux-modules-extra.*4.15.0-38-generic|^linux-signed-image.*4.15.0-38-generic|^kfreebsd-image.*4.15.0-38-generic|^kfreebsd-headers.*4.15.0-38-generic|^gnumach-image.*4.15.0-38-generic|4.15.0-38-generic.*-modules|4.15.0-38-generic.*-kernel|^linux-backports-modules-.*.*4.15.0-38-generic|^linux-modules-.*.*4.15.0-38-generic|^linux-tools.*4.15.0-38-generic|^linux-cloud-tools.*4.15.0-38-generic) regexp to find running kernel packages
  Checking: apport ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>])
  adjusting candidate version: apport=2.20.9-0ubuntu7.1
  ...
  Checking: libnss-systemd ([<Origin component:'main' archive:'bionic-updates' origin:'Ubuntu' label:'Ubuntu' site:'archive.ubuntu.com' isTrusted:True>, <Origin component:'main' archive:'bionic-security' origin:'Ubuntu' label:'Ubuntu' site:'security.ubuntu.com' isTrusted:True>])
  skipping blacklisted package systemd
  pkg systemd package has been blacklisted
  sanity check failed
  ['libfreetype6']
  []
  ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python-apt/+bug/1803749/+subscriptions