[Bug 1877769] Re: [SRU] Rewinding cache triggers obsolete adjustments consuming a lot of CPU

[Balint Reczey]

Verified 2.3ubuntu0.1 on Focal:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac
/autopkgtest-focal/focal/amd64/u/unattended-
upgrades/20200722_172955_59a0f@/log.gz :

...
Running ./test_rewind.py with python3
INFO:root:Allowed origins are: o=Ubuntu,a=lucid-security
INFO:root:Initial blacklist: ant-doc
INFO:root:Initial whitelist (not strict): 
DEBUG:root:Marking not allowed <apt_pkg.PackageFile object: filename:'/tmp/autopkgtest.QeV87E/build.hoM/src/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0> with -32768 pin
DEBUG:root:Applying pinning: PkgFilePin(id=0, priority=-32768)
DEBUG:root:Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/tmp/autopkgtest.QeV87E/build.hoM/src/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0>
DEBUG:root:APT::VersionedKernelPackages is not set
DEBUG:root:adjusting candidate version: test-package=2.0
DEBUG:root:adjusting candidate version: test2-package=2.0
DEBUG:root:falling back to adjusting test2-package's dependencies
DEBUG:root:falling back to adjusting z-package's dependencies
DEBUG:root:sanity check failed for: set() : no package is selected to be upgraded or installed
.
----------------------------------------------------------------------
Ran 1 test in 0.063s

OK
...

** Tags removed: verification-needed verification-needed-focal
** Tags added: verification-done verification-done-focal

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1877769

Title:
  [SRU] Rewinding cache triggers obsolete adjustments consuming a lot of
  CPU

Status in unattended-upgrades package in Ubuntu:
  Fix Released
Status in unattended-upgrades source package in Focal:
  Fix Committed
Status in unattended-upgrades package in Debian:
  Fix Released

Bug description:
  [Impact]

   * Unattended-upgrades consumes excessive amount of CPU when rewinding
  cache trying to mark packages to upgrade or install even when they are
  already marked to be upgraded or installed. This can be triggered by
  packages held back.

  [Test Case]

   * Run test_rewind.py:
     -Fixed output:
  $ python3 test_rewind.py 
  INFO:root:Allowed origins are: o=Ubuntu,a=lucid-security
  INFO:root:Initial blacklist: ant-doc
  INFO:root:Initial whitelist (not strict): 
  DEBUG:root:Marking not allowed <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0> with -32768 pin
  DEBUG:root:Applying pinning: PkgFilePin(id=0, priority=-32768)
  DEBUG:root:Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0>
  DEBUG:root:Applying pinning: PkgPin(pkg='/^ant-doc/', priority=-32768)
  DEBUG:root:APT::VersionedKernelPackages is not set
  DEBUG:root:adjusting candidate version: test-package=2.0
  DEBUG:root:adjusting candidate version: test2-package=2.0
  DEBUG:root:falling back to adjusting test2-package's dependencies
  DEBUG:root:falling back to adjusting z-package's dependencies
  DEBUG:root:sanity check failed for: set() : no package is selected to be upgraded or installed
  .
  ----------------------------------------------------------------------
  Ran 1 test in 0.022s

  OK

    - Not fixed output:
   $ python3 test_rewind.py 
  INFO:root:Allowed origins are: o=Ubuntu,a=lucid-security
  INFO:root:Initial blacklist: ant-doc
  INFO:root:Initial whitelist (not strict): 
  DEBUG:root:Marking not allowed <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0> with -32768 pin
  DEBUG:root:Applying pinning: PkgFilePin(id=0, priority=-32768)
  DEBUG:root:Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0>
  DEBUG:root:Applying pinning: PkgPin(pkg='/^ant-doc/', priority=-32768)
  DEBUG:root:APT::VersionedKernelPackages is not set
  DEBUG:root:adjusting candidate version: test-package=2.0
  DEBUG:root:adjusting candidate version: test2-package=2.0
  DEBUG:root:falling back to adjusting test2-package's dependencies
  DEBUG:root:falling back to adjusting z-package's dependencies
  DEBUG:root:sanity check failed for: set() : no package is selected to be upgraded or installed
  DEBUG:root:falling back to adjusting test-package's dependencies
  DEBUG:root:falling back to adjusting test2-package's dependencies
  DEBUG:root:falling back to adjusting test3-package's dependencies
  DEBUG:root:falling back to adjusting test-package's dependencies
  DEBUG:root:falling back to adjusting test2-package's dependencies
  DEBUG:root:falling back to adjusting test3-package's dependencies
  .
  ----------------------------------------------------------------------
  Ran 1 test in 0.024s

  (Note the extra "DEBUG:root:falling back to adjusting" ... lines)

  [Regression Potential]

   * Minimal. The very small change fixes how call_checked() verifies that the package to be upgrades/installed is indeed marked to either being installed or upgraded.
  In the worst case, if call_checked() erroneously returns success for a package as a result of a regression, unattended-upgrades does not fall back to adjusting packages to make marking successful. Later unattended-upgrades finds out that the package could not be marked and should be treated as kept back.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1877769/+subscriptions