[Bug 1874272] Re: Stage "searching for obsolete software" takes a very long time (30 minutes)

[Julian Andres Klode]

Andy, the issue re-occurs with 24.04.23 if you upgrade from an out-of-
date jammy system that did not have apt 2.4.13 installed prior to
starting the upgrade, or 23.10 (as we can't fix apt there), as we needed
to add a fall-back to the old algorithm in that version in those cases.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1874272

Title:
  Stage "searching for obsolete software" takes a very long time (30
  minutes)

Status in ubuntu-release-upgrader package in Ubuntu:
  Fix Released
Status in ubuntu-release-upgrader source package in Focal:
  Won't Fix
Status in ubuntu-release-upgrader source package in Noble:
  Fix Released

Bug description:
  [Impact]
  Upgrades hang for a long time searching for obsolete software, frustrating users.

  [Test plan]

  1. With each, update, and proposed version do:

    0. launch a container / VM
    1. do-release-upgrade
    2. Note down the time it took for the obsolete software stage, and the packages marked for removal.

  2. Check that the time is now significantly smaller than before

  For example with the default ubuntu:jammy lxd container:

  before: ~8s
  2024-07-16 15:13:25,575 DEBUG Start checking for obsolete pkgs
  2024-07-16 15:13:33,847 DEBUG Finish checking for obsolete pkgs

  after: ~0.5s
  2024-07-16 15:22:21,977 DEBUG Start checking for obsolete pkgs
  2024-07-16 15:22:22,508 DEBUG Finish checking for obsolete pkgs

  3. Compare the list of packages marked for removal: It should be the
  same or more packages.

  before:
  2024-07-16 15:13:33,907 DEBUG The following packages are marked for removal: libperl5.34 python3.10 libctf-nobfd0 libblockdev-part2 libblockdev-swap2 libpcre3 libunistring2 libaio1 libmpdec3 libbinutils python3-jeepney libpython3.10-minimal libblockdev-fs2 libpython3.10-stdlib binuti
  ls-x86-64-linux-gnu irqbalance libbpf0 libtss2-mu0 libldap-2.5-0 isc-dhcp-common libisc-export1105 libappstream4 libpython3.10 libblockdev-loop2 libblockdev2 libblockdev-part-err2 isc-dhcp-client libctf0 libnsl2 libblockdev-utils2 python3-jaraco.classes libprocps8 binutils-common libdns-export1110 libsframe1 pkexec python3-keyring libgprofng0 libcbor0.8 python3.10-minimal perl-modules-5.34 ubuntu-advantage-tools gcc-12-base libblockdev-crypto2 libicu70 libnetplan0 policykit-1 binutils python3-secretstorage                                                         

  after:
  2024-07-16 15:22:22,555 DEBUG The following packages are marked for removal: libperl5.34 python3.10 libctf-nobfd0 libblockdev-part2 libblockdev-swap2 libpcre3 libunistring2 libaio1 libmpdec3 libbinutils python3-jeepney libpython3.10-minimal libblockdev-fs2 libpython3.10-stdlib binuti
  ls-x86-64-linux-gnu irqbalance libbpf0 libtss2-mu0 libldap-2.5-0 isc-dhcp-common libisc-export1105 libappstream4 libpython3.10 libblockdev-loop2 libblockdev2 libblockdev-part-err2 isc-dhcp-client libctf0 libnsl2 libblockdev-utils2 python3-jaraco.classes libprocps8 binutils-common libdns-export1110 libsframe1 pkexec python3-keyring libgprofng0 libcbor0.8 python3.10-minimal perl-modules-5.34 ubuntu-advantage-tools gcc-12-base libblockdev-crypto2 libicu70 libnetplan0 policykit-1 binutils python3-secretstorage

  
  This small container is not necessarily the best test case to demonstrate it and ensure correctness but the fastest one; it may also be suitable to run it in a desktop VM instead.

  [Where problems could occur]
  The upgrade process could crash in case the solver fails to solve the request it is being given. This is the last step of the upgrade though, so it is reasonable safe.

  One could have skipped on a solver failure but this way we'd get crash reports
  in the unlikely case they happen.

  [Original bug report]

  
  Test Case:
  1. Fresh Bionic installation from 18.04.4 + all updates applied
  2. Run: update-manager -d
  3. Proceed with the upgrade.

  Actual Result:
  Everything goes well but the stage "Searching for obsolete software" takes a very long time.

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  Package: ubuntu-release-upgrader-core 1:20.04.18
  ProcVersionSignature: Ubuntu 5.4.0-26.30-generic 5.4.30
  Uname: Linux 5.4.0-26-generic x86_64
  ApportVersion: 2.20.11-0ubuntu27
  Architecture: amd64
  CasperMD5CheckResult: skip
  CrashDB: ubuntu
  CurrentDesktop: ubuntu:GNOME
  Date: Wed Apr 22 10:34:19 2020
  InstallationDate: Installed on 2020-04-20 (2 days ago)
  InstallationMedia: Ubuntu 18.04.4 LTS "Bionic Beaver" - Release amd64 (20200203.1)
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm-256color
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: ubuntu-release-upgrader
  Symptom: ubuntu-release-upgrader
  UpgradeStatus: Upgraded to focal on 2020-04-22 (0 days ago)
  VarLogDistupgradeTermlog:

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1874272/+subscriptions