[Bug 2083975] [NEW] Cephadm on Server 24.04.1 arm64 pulls wrong package

Tue Oct 8 20:12:40 UTC 2024

Public bug reported:

What I was doing:
I tried to bootstrap a Ceph cluster on an Ubuntu Server 24.04.1 arm64 with cephadm installed by apt from the Ubuntu repository.

What happened:
cephadm tries to pull the x86_64/amd64 version of the package instead of the arm64 version, resulting in an exec format error. 

What I expected to happen:
cephadm should pull the correct architecture of package from the repositiory.

Steps to reproduce:
Install unmodified Ubuntu 24.04.1 LTS on an arm64 platform without Docker (!)
Follow instructions from ceph documentation to bootstrap cluster with cephadm
Observe the error
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
ERROR (catatonit:2): failed to exec pid1: Exec format error

Workaround:
Force correct image utilization:
sudo cephadm --image quay.ceph.io/ceph-ci/ceph:main-aarch64 bootstrap --mon-ip 10.211.55.25 --cluster-network 10.37.133.0/24
Observe correct execution of the bootstrap and subsequently a functional cluster

Additional information:
I did file a bug with Ceph https://tracker.ceph.com/issues/68240, but was advised that while the packages from quay.io are expected to automatically pull the right architecture, the packages from quay.ceph.io do not (due to some unspecified internal decision), so what Ubuntu Server 24.04.1 cephadm is doing is not expected to work according to the Ceph team.

Docker must be uninstalled, because the image supplied by quay.ceph.io
only works with podman. With docker the correct image is pulled, but
install crashes when the cluster tries to write /tmp/monmap with the
error /tmp/monmap is a directory. With podman it works correctly.

I have a working cluster with the workaround, but in my opinion this
should work out of the box. It took me a considerable amount of time to
find out what image I have to specify to make this work.

Version of Ubuntu: Ubuntu 24.04.1 LTS
Version of cephadm as seen by cephadm version: UNKNOWN (this is also incorrect)
Version of cephadm as seen by dpkg -l: 19.2.0~git20240301.4c76c50-0ubuntu6.1 
Version of cephadm as seen by apt: cephadm/noble-updates,now 19.2.0~git20240301.4c76c50-0ubuntu6.1 arm64
Version of docker (that fails): docker.io/noble-updates 24.0.7-0ubuntu4.1 arm64
Version of podman (that works): podman/noble-updates,noble-security,now 4.9.3+ds1-1ubuntu0.1 arm64

Note: There is also a bug in the bugtracker. The package cephadm does exist in Ubuntu, but it won't let me file a bug against it with the error ""cephadm" does not exist in Ubuntu. Please choose a different package. If you're unsure, please select "I don't know"
=

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: cephadm 19.2.0~git20240301.4c76c50-0ubuntu6.1
ProcVersionSignature: Ubuntu 6.8.0-45.45-generic 6.8.12
Uname: Linux 6.8.0-45-generic aarch64
ApportVersion: 2.28.1-0ubuntu3.1
Architecture: arm64
CasperMD5CheckResult: pass
Date: Tue Oct  8 19:50:43 2024
InstallationDate: Installed on 2024-10-08 (0 days ago)
InstallationMedia: Ubuntu-Server 24.04.1 LTS "Noble Numbat" - Release arm64 (20240827)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: ceph
UpgradeStatus: No upgrade log present (probably fresh install)

** Affects: ceph (Ubuntu)
     Importance: Undecided
         Status: New

** Tags: apport-bug arm64 noble

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2083975

Title:
  Cephadm on Server 24.04.1 arm64 pulls wrong package

Status in ceph package in Ubuntu:
  New

Bug description:
  What I was doing:
  I tried to bootstrap a Ceph cluster on an Ubuntu Server 24.04.1 arm64 with cephadm installed by apt from the Ubuntu repository.

  What happened:
  cephadm tries to pull the x86_64/amd64 version of the package instead of the arm64 version, resulting in an exec format error. 

  What I expected to happen:
  cephadm should pull the correct architecture of package from the repositiory.

  Steps to reproduce:
  Install unmodified Ubuntu 24.04.1 LTS on an arm64 platform without Docker (!)
  Follow instructions from ceph documentation to bootstrap cluster with cephadm
  Observe the error
  WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
  ERROR (catatonit:2): failed to exec pid1: Exec format error

  Workaround:
  Force correct image utilization:
  sudo cephadm --image quay.ceph.io/ceph-ci/ceph:main-aarch64 bootstrap --mon-ip 10.211.55.25 --cluster-network 10.37.133.0/24
  Observe correct execution of the bootstrap and subsequently a functional cluster

  Additional information:
  I did file a bug with Ceph https://tracker.ceph.com/issues/68240, but was advised that while the packages from quay.io are expected to automatically pull the right architecture, the packages from quay.ceph.io do not (due to some unspecified internal decision), so what Ubuntu Server 24.04.1 cephadm is doing is not expected to work according to the Ceph team.

  Docker must be uninstalled, because the image supplied by quay.ceph.io
  only works with podman. With docker the correct image is pulled, but
  install crashes when the cluster tries to write /tmp/monmap with the
  error /tmp/monmap is a directory. With podman it works correctly.

  I have a working cluster with the workaround, but in my opinion this
  should work out of the box. It took me a considerable amount of time
  to find out what image I have to specify to make this work.

  Version of Ubuntu: Ubuntu 24.04.1 LTS
  Version of cephadm as seen by cephadm version: UNKNOWN (this is also incorrect)
  Version of cephadm as seen by dpkg -l: 19.2.0~git20240301.4c76c50-0ubuntu6.1 
  Version of cephadm as seen by apt: cephadm/noble-updates,now 19.2.0~git20240301.4c76c50-0ubuntu6.1 arm64
  Version of docker (that fails): docker.io/noble-updates 24.0.7-0ubuntu4.1 arm64
  Version of podman (that works): podman/noble-updates,noble-security,now 4.9.3+ds1-1ubuntu0.1 arm64

  Note: There is also a bug in the bugtracker. The package cephadm does exist in Ubuntu, but it won't let me file a bug against it with the error ""cephadm" does not exist in Ubuntu. Please choose a different package. If you're unsure, please select "I don't know"
  =

  ProblemType: Bug
  DistroRelease: Ubuntu 24.04
  Package: cephadm 19.2.0~git20240301.4c76c50-0ubuntu6.1
  ProcVersionSignature: Ubuntu 6.8.0-45.45-generic 6.8.12
  Uname: Linux 6.8.0-45-generic aarch64
  ApportVersion: 2.28.1-0ubuntu3.1
  Architecture: arm64
  CasperMD5CheckResult: pass
  Date: Tue Oct  8 19:50:43 2024
  InstallationDate: Installed on 2024-10-08 (0 days ago)
  InstallationMedia: Ubuntu-Server 24.04.1 LTS "Noble Numbat" - Release arm64 (20240827)
  ProcEnviron:
   LANG=en_US.UTF-8
   PATH=(custom, no user)
   SHELL=/bin/bash
   TERM=xterm-256color
   XDG_RUNTIME_DIR=<set>
  SourcePackage: ceph
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2083975/+subscriptions