[Bug 1877769] Re: [SRU] Rewinding cache triggers obsolete adjustments consuming a lot of CPU

Balint Reczey balint.reczey at canonical.com
Mon May 11 13:50:32 UTC 2020 

** Description changed:

- TODO
+ [Impact]
+ 
+  * Unattended-upgrades consumes excessive amount of CPU when rewinding
+ cache trying to mark packages to upgrade or install even when they are
+ already marked to be upgraded or installed. This can be triggered by
+ packages held back.
+ 
+ [Test Case]
+ 
+  * Run test_rewind.py:
+    -Fixed output:
+ $ python3 test_rewind.py 
+ INFO:root:Allowed origins are: o=Ubuntu,a=lucid-security
+ INFO:root:Initial blacklist: ant-doc
+ INFO:root:Initial whitelist (not strict): 
+ DEBUG:root:Marking not allowed <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0> with -32768 pin
+ DEBUG:root:Applying pinning: PkgFilePin(id=0, priority=-32768)
+ DEBUG:root:Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0>
+ DEBUG:root:Applying pinning: PkgPin(pkg='/^ant-doc/', priority=-32768)
+ DEBUG:root:APT::VersionedKernelPackages is not set
+ DEBUG:root:adjusting candidate version: test-package=2.0
+ DEBUG:root:adjusting candidate version: test2-package=2.0
+ DEBUG:root:falling back to adjusting test2-package's dependencies
+ DEBUG:root:falling back to adjusting z-package's dependencies
+ DEBUG:root:sanity check failed for: set() : no package is selected to be upgraded or installed
+ .
+ ----------------------------------------------------------------------
+ Ran 1 test in 0.022s
+ 
+ OK
+ 
+   - Not fixed output:
+  $ python3 test_rewind.py 
+ INFO:root:Allowed origins are: o=Ubuntu,a=lucid-security
+ INFO:root:Initial blacklist: ant-doc
+ INFO:root:Initial whitelist (not strict): 
+ DEBUG:root:Marking not allowed <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0> with -32768 pin
+ DEBUG:root:Applying pinning: PkgFilePin(id=0, priority=-32768)
+ DEBUG:root:Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0>
+ DEBUG:root:Applying pinning: PkgPin(pkg='/^ant-doc/', priority=-32768)
+ DEBUG:root:APT::VersionedKernelPackages is not set
+ DEBUG:root:adjusting candidate version: test-package=2.0
+ DEBUG:root:adjusting candidate version: test2-package=2.0
+ DEBUG:root:falling back to adjusting test2-package's dependencies
+ DEBUG:root:falling back to adjusting z-package's dependencies
+ DEBUG:root:sanity check failed for: set() : no package is selected to be upgraded or installed
+ DEBUG:root:falling back to adjusting test-package's dependencies
+ DEBUG:root:falling back to adjusting test2-package's dependencies
+ DEBUG:root:falling back to adjusting test3-package's dependencies
+ DEBUG:root:falling back to adjusting test-package's dependencies
+ DEBUG:root:falling back to adjusting test2-package's dependencies
+ DEBUG:root:falling back to adjusting test3-package's dependencies
+ .
+ ----------------------------------------------------------------------
+ Ran 1 test in 0.024s
+ 
+ (Note the extra "DEBUG:root:falling back to adjusting" ... lines)
+ 
+ [Regression Potential]
+ 
+  * Minimal. The very small change fixes how call_checked() verifies that the package to be upgrades/installed is indeed marked to either being installed or upgraded.
+ In the worst case, if call_checked() erroneously returns success for a package as a result of a regression, unattended-upgrades does not fall back to adjusting packages to make marking successful. Later unattended-upgrades finds out that the package could not be marked and should be treated as kept back.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to unattended-upgrades in Ubuntu.
https://bugs.launchpad.net/bugs/1877769

Title:
  [SRU] Rewinding cache triggers obsolete adjustments consuming a lot of
  CPU

Status in unattended-upgrades package in Ubuntu:
  In Progress
Status in unattended-upgrades package in Debian:
  Confirmed

Bug description:
  [Impact]

   * Unattended-upgrades consumes excessive amount of CPU when rewinding
  cache trying to mark packages to upgrade or install even when they are
  already marked to be upgraded or installed. This can be triggered by
  packages held back.

  [Test Case]

   * Run test_rewind.py:
     -Fixed output:
  $ python3 test_rewind.py 
  INFO:root:Allowed origins are: o=Ubuntu,a=lucid-security
  INFO:root:Initial blacklist: ant-doc
  INFO:root:Initial whitelist (not strict): 
  DEBUG:root:Marking not allowed <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0> with -32768 pin
  DEBUG:root:Applying pinning: PkgFilePin(id=0, priority=-32768)
  DEBUG:root:Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0>
  DEBUG:root:Applying pinning: PkgPin(pkg='/^ant-doc/', priority=-32768)
  DEBUG:root:APT::VersionedKernelPackages is not set
  DEBUG:root:adjusting candidate version: test-package=2.0
  DEBUG:root:adjusting candidate version: test2-package=2.0
  DEBUG:root:falling back to adjusting test2-package's dependencies
  DEBUG:root:falling back to adjusting z-package's dependencies
  DEBUG:root:sanity check failed for: set() : no package is selected to be upgraded or installed
  .
  ----------------------------------------------------------------------
  Ran 1 test in 0.022s

  OK

    - Not fixed output:
   $ python3 test_rewind.py 
  INFO:root:Allowed origins are: o=Ubuntu,a=lucid-security
  INFO:root:Initial blacklist: ant-doc
  INFO:root:Initial whitelist (not strict): 
  DEBUG:root:Marking not allowed <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0> with -32768 pin
  DEBUG:root:Applying pinning: PkgFilePin(id=0, priority=-32768)
  DEBUG:root:Applying pin -32768 to package_file: <apt_pkg.PackageFile object: filename:'/home/rbalint/projects/deb/unattended-upgrades/test/root.rewind/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages'  a=lucid-updates,c=main,v=10.04,o=Ubuntu,l=Ubuntu arch='amd64' site='archive.ubuntu.com' IndexType='Debian Package Index' Size=894 ID:0>
  DEBUG:root:Applying pinning: PkgPin(pkg='/^ant-doc/', priority=-32768)
  DEBUG:root:APT::VersionedKernelPackages is not set
  DEBUG:root:adjusting candidate version: test-package=2.0
  DEBUG:root:adjusting candidate version: test2-package=2.0
  DEBUG:root:falling back to adjusting test2-package's dependencies
  DEBUG:root:falling back to adjusting z-package's dependencies
  DEBUG:root:sanity check failed for: set() : no package is selected to be upgraded or installed
  DEBUG:root:falling back to adjusting test-package's dependencies
  DEBUG:root:falling back to adjusting test2-package's dependencies
  DEBUG:root:falling back to adjusting test3-package's dependencies
  DEBUG:root:falling back to adjusting test-package's dependencies
  DEBUG:root:falling back to adjusting test2-package's dependencies
  DEBUG:root:falling back to adjusting test3-package's dependencies
  .
  ----------------------------------------------------------------------
  Ran 1 test in 0.024s

  (Note the extra "DEBUG:root:falling back to adjusting" ... lines)

  [Regression Potential]

   * Minimal. The very small change fixes how call_checked() verifies that the package to be upgrades/installed is indeed marked to either being installed or upgraded.
  In the worst case, if call_checked() erroneously returns success for a package as a result of a regression, unattended-upgrades does not fall back to adjusting packages to make marking successful. Later unattended-upgrades finds out that the package could not be marked and should be treated as kept back.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1877769/+subscriptions

More information about the foundations-bugs mailing list