[Bug 2150762] Re: Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3 subinterpreter incompatibility)

[Luciano Lo Giudice]

This bug is now being handled in the SRU for 20.2.1:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2151199

[ Impact ]

 * cephadm is unusable in Ubuntu Resolute, due to issues with sub-interpreters
   in the shipped Python version (3.12).
   
 * This problem is very similar to what happened in the initial Ceph packages
   in Noble (19.2.0), and the fix is very similar in consequence. The proposed
   fix consists of backporting some commits done by upstream Ceph developers
   that have landed in the 'main' branch, but not yet in any stable release.

[ Test Plan ]

 * In order to test this, the functional tests for Canonical's Ceph containers
   repository will be run (https://github.com/canonical/ceph-containers).
   
 * These tests involve using the cephadm tool for cluster deployment using
   Canonical's custom Ceph images.
   
 * If the new packages fix this issue, the tests will run to completion.

[ Where problems could occur ]

 * cephadm is used as a simple tool to deploy Ceph clusters. Notably, neither
   the Ceph charms nor Microceph make use of it, but users of Ubuntu can make
   use of it independently.
   
 * Any problems with cephadm will thus prevent users from deploying their own
   Ceph clusters with the exceptions noted above.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2150762

Title:
  Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3
  subinterpreter incompatibility)

Status in ceph package in Ubuntu:
  New

Bug description:
  Package: ceph 20.2.0-0ubuntu2 (resolute)
  Related: python3-cryptography 46.0.5-1ubuntu2

  On resolute (Ubuntu 26.04, Python 3.14.4), `cephadm bootstrap` aborts at
  the `mgr module enable cephadm` step with:

    Error ENOENT: module 'cephadm' reports that it cannot run on the active
    manager daemon: PyO3 modules do not yet support subinterpreters, see
    https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

  Reproduced in an LXD VM running cephadm 20.2.0 from resolute, bootstrapping
  against a rock built from the same archive. The mgr daemon log
  (via `journalctl -u 'ceph-*@mgr.<host>.*'`) shows the actual Python
  traceback when the cephadm module is loaded in a subinterpreter:

    File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
        from .module import CephadmOrchestrator
    File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
        from cephadm.cert_mgr import CertMgr
    File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
        from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
    File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
        from cryptography import x509
    File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
        in <module>
        from cryptography.x509 import certificate_transparency, verification
    File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
        line 8, in <module>
        from cryptography.hazmat.bindings._rust import x509 as rust_x509
    ImportError: PyO3 modules do not yet support subinterpreters,
        see https://github.com/PyO3/pyo3/issues/576

  The same failure mode applies to the diskprediction_local mgr module,
  where the failing import is `numpy._core._multiarray_umath` (also
  PyO3-built in resolute's python3-numpy 2.3.x).

  Root cause:
  - Tentacle (20.2.x) ceph-mgr loads each mgr module in a separate Python
    subinterpreter by default.
  - Resolute's python3-cryptography 46.0.5 includes Rust extensions built
    with PyO3, which raises ImportError when imported into a subinterpreter
    (PyO3 currently does not support PEP 684 multi-interpreter modules).

  Upstream fix:
  ceph/ceph commit 239b0dc8 ("mgr: add mgr_subinterpreter_modules config",
  merged 2025-11-07) inverts the default: mgr modules load in the main
  interpreter, and `mgr_subinterpreter_modules` is an opt-in list. The fix
  is in master but is NOT in the v20.2.0 or v20.2.1 release tags (verified
  by reading src/common/options/mgr.yaml.in at both tags). It is therefore
  absent from resolute's ceph 20.2.0-0ubuntu2.

  Requested action:
  Backport ceph/ceph commit 239b0dc8 into resolute's ceph package so
  `mgr module enable cephadm` succeeds on resolute. Until that lands,
  canonical/ceph-containers's Tentacle rock works around this by switching
  back to noble base + ppa:lmlogiudice/ceph-tentacle-noble (the same source
  MicroCeph uses).

  Reproduction (LXD VM):
    1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
    2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
    3. cephadm --image <resolute-based-tentacle-OCI> bootstrap --mon-ip <ip> \
         --single-host-defaults --skip-dashboard --skip-monitoring-stack
    4. Observe the PyO3 error during the `mgr module enable cephadm` step.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2150762/+subscriptions