[Bug 2150762] Re: Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3 subinterpreter incompatibility)

[Luciano Lo Giudice]

** Description changed:

  [ Impact ]
  
-  * cephadm is unusable in Ubuntu Resolute, due to issues with sub-interpreters
-    in the shipped Python version (3.12).
+  * cephadm is unusable in Ubuntu Resolute, due to issues with sub-interpreters
+    in the shipped Python version (3.14)
  
-  * This problem is very similar to what happened in the initial Ceph packages
-    in Noble (19.2.0), and the fix is very similar in consequence. The proposed
-    fix consists of backporting some commits done by upstream Ceph developers
-    that have landed in the 'main' branch, but not yet in any stable release.
+  * This problem is very similar to what happened in the initial Ceph packages
+    in Noble (19.2.0), and the fix is very similar in consequence. The proposed
+    fix consists of backporting some commits done by upstream Ceph developers
+    that have landed in the 'main' branch, but not yet in any stable release.
  
  [ Test Plan ]
  
-  * In order to test this, the functional tests for Canonical's Ceph containers
-    repository will be run (https://github.com/canonical/ceph-containers).
+  * In order to test this, the functional tests for Canonical's Ceph containers
+    repository will be run (https://github.com/canonical/ceph-containers).
  
-  * These tests involve using the cephadm tool for cluster deployment using
-    Canonical's custom Ceph images.
+  * These tests involve using the cephadm tool for cluster deployment using
+    Canonical's custom Ceph images.
  
-  * If the new packages fix this issue, the tests will run to completion.
+  * If the new packages fix this issue, the tests will run to completion.
  
  [ Where problems could occur ]
  
-  * cephadm is used as a simple tool to deploy Ceph clusters. Notably, neither
-    the Ceph charms nor Microceph make use of it, but users of Ubuntu can make
-    use of it independently.
+  * cephadm is used as a simple tool to deploy Ceph clusters. Notably, neither
+    the Ceph charms nor Microceph make use of it, but users of Ubuntu can make
+    use of it independently.
  
-  * Any problems with cephadm will thus prevent users from deploying their own
-    Ceph clusters with the exceptions noted above.
-    
+  * Any problems with cephadm will thus prevent users from deploying their own
+    Ceph clusters with the exceptions noted above.
+ 
  [ Original description ]
  
  Package: ceph 20.2.0-0ubuntu2 (resolute)
  Related: python3-cryptography 46.0.5-1ubuntu2
  
  On resolute (Ubuntu 26.04, Python 3.14.4), `cephadm bootstrap` aborts at
  the `mgr module enable cephadm` step with:
  
-   Error ENOENT: module 'cephadm' reports that it cannot run on the active
-   manager daemon: PyO3 modules do not yet support subinterpreters, see
-   https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)
+   Error ENOENT: module 'cephadm' reports that it cannot run on the active
+   manager daemon: PyO3 modules do not yet support subinterpreters, see
+   https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)
  
  Reproduced in an LXD VM running cephadm 20.2.0 from resolute, bootstrapping
  against a rock built from the same archive. The mgr daemon log
  (via `journalctl -u 'ceph-*@mgr.<host>.*'`) shows the actual Python
  traceback when the cephadm module is loaded in a subinterpreter:
  
-   File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
-       from .module import CephadmOrchestrator
-   File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
-       from cephadm.cert_mgr import CertMgr
-   File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
-       from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
-   File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
-       from cryptography import x509
-   File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
-       in <module>
-       from cryptography.x509 import certificate_transparency, verification
-   File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
-       line 8, in <module>
-       from cryptography.hazmat.bindings._rust import x509 as rust_x509
-   ImportError: PyO3 modules do not yet support subinterpreters,
-       see https://github.com/PyO3/pyo3/issues/576
+   File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
+       from .module import CephadmOrchestrator
+   File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
+       from cephadm.cert_mgr import CertMgr
+   File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
+       from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
+   File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
+       from cryptography import x509
+   File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
+       in <module>
+       from cryptography.x509 import certificate_transparency, verification
+   File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
+       line 8, in <module>
+       from cryptography.hazmat.bindings._rust import x509 as rust_x509
+   ImportError: PyO3 modules do not yet support subinterpreters,
+       see https://github.com/PyO3/pyo3/issues/576
  
  The same failure mode applies to the diskprediction_local mgr module,
  where the failing import is `numpy._core._multiarray_umath` (also
  PyO3-built in resolute's python3-numpy 2.3.x).
  
  Root cause:
  - Tentacle (20.2.x) ceph-mgr loads each mgr module in a separate Python
-   subinterpreter by default.
+   subinterpreter by default.
  - Resolute's python3-cryptography 46.0.5 includes Rust extensions built
-   with PyO3, which raises ImportError when imported into a subinterpreter
-   (PyO3 currently does not support PEP 684 multi-interpreter modules).
+   with PyO3, which raises ImportError when imported into a subinterpreter
+   (PyO3 currently does not support PEP 684 multi-interpreter modules).
  
  Upstream fix:
  ceph/ceph commit 239b0dc8 ("mgr: add mgr_subinterpreter_modules config",
  merged 2025-11-07) inverts the default: mgr modules load in the main
  interpreter, and `mgr_subinterpreter_modules` is an opt-in list. The fix
  is in master but is NOT in the v20.2.0 or v20.2.1 release tags (verified
  by reading src/common/options/mgr.yaml.in at both tags). It is therefore
  absent from resolute's ceph 20.2.0-0ubuntu2.
  
  Requested action:
  Backport ceph/ceph commit 239b0dc8 into resolute's ceph package so
  `mgr module enable cephadm` succeeds on resolute. Until that lands,
  canonical/ceph-containers's Tentacle rock works around this by switching
  back to noble base + ppa:lmlogiudice/ceph-tentacle-noble (the same source
  MicroCeph uses).
  
  Reproduction (LXD VM):
-   1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
-   2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
-   3. cephadm --image <resolute-based-tentacle-OCI> bootstrap --mon-ip <ip> \
-        --single-host-defaults --skip-dashboard --skip-monitoring-stack
-   4. Observe the PyO3 error during the `mgr module enable cephadm` step.
+   1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
+   2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
+   3. cephadm --image <resolute-based-tentacle-OCI> bootstrap --mon-ip <ip> \
+        --single-host-defaults --skip-dashboard --skip-monitoring-stack
+   4. Observe the PyO3 error during the `mgr module enable cephadm` step.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2150762

Title:
  Ceph 20.2.0 mgr cephadm module fails to load on resolute (PyO3
  subinterpreter incompatibility)

Status in ceph package in Ubuntu:
  In Progress
Status in ceph source package in Resolute:
  In Progress
Status in ceph source package in Stonking:
  In Progress

Bug description:
  [ Impact ]

   * cephadm is unusable in Ubuntu Resolute, due to issues with sub-interpreters
     in the shipped Python version (3.14)

   * This problem is very similar to what happened in the initial Ceph packages
     in Noble (19.2.0), and the fix is very similar in consequence. The proposed
     fix consists of backporting some commits done by upstream Ceph developers
     that have landed in the 'main' branch, but not yet in any stable release.

  [ Test Plan ]

   * In order to test this, the functional tests for Canonical's Ceph containers
     repository will be run (https://github.com/canonical/ceph-containers).

   * These tests involve using the cephadm tool for cluster deployment using
     Canonical's custom Ceph images.

   * If the new packages fix this issue, the tests will run to
  completion.

  [ Where problems could occur ]

   * cephadm is used as a simple tool to deploy Ceph clusters. Notably, neither
     the Ceph charms nor Microceph make use of it, but users of Ubuntu can make
     use of it independently.

   * Any problems with cephadm will thus prevent users from deploying their own
     Ceph clusters with the exceptions noted above.

  [ Original description ]

  Package: ceph 20.2.0-0ubuntu2 (resolute)
  Related: python3-cryptography 46.0.5-1ubuntu2

  On resolute (Ubuntu 26.04, Python 3.14.4), `cephadm bootstrap` aborts at
  the `mgr module enable cephadm` step with:

    Error ENOENT: module 'cephadm' reports that it cannot run on the active
    manager daemon: PyO3 modules do not yet support subinterpreters, see
    https://github.com/PyO3/pyo3/issues/576 (pass --force to force enablement)

  Reproduced in an LXD VM running cephadm 20.2.0 from resolute, bootstrapping
  against a rock built from the same archive. The mgr daemon log
  (via `journalctl -u 'ceph-*@mgr.<host>.*'`) shows the actual Python
  traceback when the cephadm module is loaded in a subinterpreter:

    File "/usr/share/ceph/mgr/cephadm/__init__.py", line 1, in <module>
        from .module import CephadmOrchestrator
    File "/usr/share/ceph/mgr/cephadm/module.py", line 19, in <module>
        from cephadm.cert_mgr import CertMgr
    File "/usr/share/ceph/mgr/cephadm/cert_mgr.py", line 4, in <module>
        from cephadm.ssl_cert_utils import SSLCerts, SSLConfigException
    File "/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py", line 6, in <module>
        from cryptography import x509
    File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7,
        in <module>
        from cryptography.x509 import certificate_transparency, verification
    File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py",
        line 8, in <module>
        from cryptography.hazmat.bindings._rust import x509 as rust_x509
    ImportError: PyO3 modules do not yet support subinterpreters,
        see https://github.com/PyO3/pyo3/issues/576

  The same failure mode applies to the diskprediction_local mgr module,
  where the failing import is `numpy._core._multiarray_umath` (also
  PyO3-built in resolute's python3-numpy 2.3.x).

  Root cause:
  - Tentacle (20.2.x) ceph-mgr loads each mgr module in a separate Python
    subinterpreter by default.
  - Resolute's python3-cryptography 46.0.5 includes Rust extensions built
    with PyO3, which raises ImportError when imported into a subinterpreter
    (PyO3 currently does not support PEP 684 multi-interpreter modules).

  Upstream fix:
  ceph/ceph commit 239b0dc8 ("mgr: add mgr_subinterpreter_modules config",
  merged 2025-11-07) inverts the default: mgr modules load in the main
  interpreter, and `mgr_subinterpreter_modules` is an opt-in list. The fix
  is in master but is NOT in the v20.2.0 or v20.2.1 release tags (verified
  by reading src/common/options/mgr.yaml.in at both tags). It is therefore
  absent from resolute's ceph 20.2.0-0ubuntu2.

  Requested action:
  Backport ceph/ceph commit 239b0dc8 into resolute's ceph package so
  `mgr module enable cephadm` succeeds on resolute. Until that lands,
  canonical/ceph-containers's Tentacle rock works around this by switching
  back to noble base + ppa:lmlogiudice/ceph-tentacle-noble (the same source
  MicroCeph uses).

  Reproduction (LXD VM):
    1. lxc launch --vm ubuntu:26.04 r1 -c limits.cpu=4 -c limits.memory=8GiB
    2. apt install cephadm openssh-server jq python3-ceph-common ceph-common
    3. cephadm --image <resolute-based-tentacle-OCI> bootstrap --mon-ip <ip> \
         --single-host-defaults --skip-dashboard --skip-monitoring-stack
    4. Observe the PyO3 error during the `mgr module enable cephadm` step.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2150762/+subscriptions