[Bug 1960065] Re: cloud archive: apt upgrade from rocky to stein keeps back some ceph packages

Fri Feb 4 20:31:43 UTC 2022

Hi Mauricio,

I just stepped through the following successfully. Is this limited to
using 'apt upgrade'?

root at bionic-uca:~# history
    1  sudo apt --yes purge unattended-upgrades
    2  apt update
    3  sudo apt --yes install ceph
    4  dpkg -s ceph | grep Version
    5  sudo add-apt-repository --yes cloud-archive:rocky
    6  man apt
    7  sudo apt dist-upgrade
    8  dpkg -s ceph | grep Version
    9  sudo add-apt-repository --yes cloud-archive:stein
   10  dpkg -s ceph | grep Version
   11  sudo apt dist-upgrade
   12  apt policy python3-ceph
   13  apt policy python3-cephfs
   14  apt policy python-cephfs

And the last 2 steps show:

root at bionic-uca:~# apt policy python3-cephfs
python3-cephfs:
  Installed: 13.2.9-0ubuntu0.19.04.1~cloud3
  Candidate: 13.2.9-0ubuntu0.19.04.1~cloud3

root at bionic-uca:~# apt policy python-cephfs
python-cephfs:
  Installed: (none)
  Candidate: 13.2.8-0ubuntu0.18.10.1~cloud0

Thanks,
Corey

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1960065

Title:
  cloud archive: apt upgrade from rocky to stein keeps back some ceph
  packages

Status in ceph package in Ubuntu:
  Confirmed

Bug description:
  UCA Rocky has older versions than UCA Stein for some Ceph python2
  packages, e.g., python-cephfs.

  This is an issue for versioned dependencies that require an exact
  version that isn't there.

  The apt solution is removing packages, but that's not allowed on `apt
  upgrade`, thus `apt` keeps back / doesn't upgrade some ceph packages
  (`apt full-upgrade` can remove/works fine.)

  Example:

  	$ apt-cache madison python-cephfs
  	python-cephfs | 13.2.8-0ubuntu0.18.10.1~cloud0 | http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
  	python-cephfs | 13.2.1+dfsg1-0ubuntu3~cloud0 | http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/stein/main amd64 Packages
  	...

  Setup Bionic container:
  ---

  $ lxc launch ubuntu:bionic mfo-bionic-uca
  $ ssh mfo-bionic-uca.lxd

  $ sudo apt --yes purge unattended-upgrades
  $ sudo apt update

  Install ceph from Bionic (Queens):
  ---

  $ sudo apt --yes install ceph

  $ dpkg -s ceph | grep Version
  Version: 12.2.13-0ubuntu0.18.04.10

  Upgrade to ceph from UCA Rocky:
  ---

  $ sudo add-apt-repository --yes cloud-archive:rocky

  $ sudo apt --dry-run upgrade
  ...
  The following packages will be upgraded:
    ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2 libradosstriper1 librbd1 python-cephfs python-pecan python-rados python-rbd
    python-webob python3-jsonpatch
  ...

  $ sudo apt --yes upgrade

  $ dpkg -s ceph | grep Version
  Version: 13.2.8-0ubuntu0.18.10.1~cloud0

  Upgrade to ceph from UCA Stein:
  ---

  $ sudo add-apt-repository --yes cloud-archive:stein

  $ sudo apt --dry-run upgrade
  ...
  The following packages have been kept back:
    ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2 libradosstriper1 librbd1
  ...

  Check:
  ---

  $ sudo apt -o Debug::pkgProblemResolver=1 --dry-run upgrade
  	...
  	Entering ResolveByKeep 10%

  	  Dependencies are not satisfied for python-cephfs:amd64 < 13.2.8-0ubuntu0.18.10.1~cloud0 @ii mK Ib >
  	Keeping package python-cephfs:amd64
  	  Dependencies are not satisfied for python-cephfs:amd64 < 13.2.8-0ubuntu0.18.10.1~cloud0 @ii mK Ib >
  	Package python-cephfs:amd64 python-cephfs:amd64 Depends on libcephfs2:amd64 < 13.2.8-0ubuntu0.18.10.1~cloud0 -> 13.2.9-0ubuntu0.19.04.1~cloud3 @ii umU > (= 13.2
  	.8-0ubuntu0.18.10.1~cloud0)
  	  Keeping Package libcephfs2:amd64 due to Depends

  ...

  Issue:
  -----

  python-cephfs at version 13.2.__8__ has exact version dep on
  libcephfs2, which is being upgraded to 13.2.__9__.

  Why isn't python-cephfs being upgraded to 13.2.__9__ too?

  AH, there's no 13.2.__9__ version for python-cephfs, only for libcephfs2.
  So, it stopped on 13.2.__8__ in UCA Rocky, there's no 13.2.__9__ in UCA Stein, as there's for libcephfs2.

  Why is that?

  OH, wait. UCA Stein does have python-cephfs, but in an _older_ version than UCA Rocky: 13.2.__1__.
  (but it has _newer_ libcephfs2, ahhh OK; there's the mismatch)

  	ubuntu at mfo-bionic-uca:~$ apt-cache madison python-cephfs
  	python-cephfs | 13.2.8-0ubuntu0.18.10.1~cloud0 | http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
  	python-cephfs | 13.2.1+dfsg1-0ubuntu3~cloud0 | http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/stein/main amd64 Packages
  	python-cephfs | 12.2.13-0ubuntu0.18.04.10 | http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
  	python-cephfs | 12.2.13-0ubuntu0.18.04.10 | http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
  	python-cephfs | 12.2.4-0ubuntu1 | http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

  	ubuntu at mfo-bionic-uca:~$ apt-cache madison libcephfs2
  	libcephfs2 | 13.2.9-0ubuntu0.19.04.1~cloud3 | http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/stein/main amd64 Packages
  	libcephfs2 | 13.2.8-0ubuntu0.18.10.1~cloud0 | http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/rocky/main amd64 Packages
  	libcephfs2 | 12.2.13-0ubuntu0.18.04.10 | http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
  	libcephfs2 | 12.2.13-0ubuntu0.18.04.10 | http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
  	libcephfs2 | 12.2.4-0ubuntu1 | http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

  Ah, the python*2* pkg is no longer built after that version:

  	Page https://launchpad.net/ubuntu/+source/ceph/13.2.1+dfsg1-0ubuntu3/+build/15590380
  	Has python3-cephfs 13.2.1+dfsg1-0ubuntu3
  	And python-cephfs 13.2.1+dfsg1-0ubuntu3

  	Page https://launchpad.net/ubuntu/+source/ceph/13.2.2+dfsg1-0ubuntu4/+build/15747822
  	Has python3-cephfs 13.2.2+dfsg1-0ubuntu4
  	NOT python-cephfs

  But how did UCA Rocky get it.
  Hm, it's from 13.2.__8__.  
  Is it an UCA only thing?

  	Yes, page not found w/out ~cloud suffix
  		https://launchpad.net/ubuntu/+source/ceph/13.2.8-0ubuntu0.18.10.1
  	Disco goes up to 13.2.7.
  		https://launchpad.net/ubuntu/disco/+source/ceph
  	Eoan starts on 13.2.4 then jumps to 14.2.1:
  		https://launchpad.net/ubuntu/eoan/+source/ceph

  So maybe do equivalently on Stein.

  And.. it's been reported that installing just `ceph-common` works, and it actually REMOVES those.

  ubuntu at mfo-bionic-uca:~$ sudo apt --dry-run install ceph-common
  ...
  The following packages will be REMOVED:
    python-cephfs python-rados python-rbd
  ...
  The following packages will be upgraded:
    ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd libcephfs2 librados2 libradosstriper1 librbd1
  ...

  And that's why `apt upgrade` doesn't work; it CANNOT remove packages.

  Ok, so `apt full-upgrade` does WORK, as it CAN:

  man 8 apt

         upgrade (apt-get(8))
             ... If an upgrade for a package
             requires the remove of an installed package the upgrade for this package isn't performed.

         full-upgrade (apt-get(8))
             full-upgrade performs the function of upgrade but will remove currently installed packages if this is needed to upgrade the system as a whole.

  And there we go.

  ubuntu at mfo-bionic-uca:~$ sudo apt --dry-run full-upgrade
  ...
  The following packages will be REMOVED:
    python-cephfs python-rados python-rbd
  The following NEW packages will be installed:
  ...
  The following packages will be upgraded:
    ceph ceph-base ceph-common ceph-mgr ceph-mon ceph-osd ibverbs-providers libcephfs2 libibverbs1 libnuma1 librados2 libradosstriper1 librbd1
    python-pkg-resources python-setuptools python-webob python3-jwt python3-pkg-resources
  ...

  Yeah, so it seems the issue is UCA Stein having older versions of some
  packages than UCA Rocky.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1960065/+subscriptions