[Bug 2154304] Re: ceph radosgw segv with keystone auth
Luciano Lo Giudice
2154304 at bugs.launchpad.net
Mon Jul 6 22:51:02 UTC 2026
** Description changed:
+ [ Impact ]
+
+ * The Ceph RADOS gateway is consistently crashing in version 20.2.0,
+ on Ubuntu resolute, when used keystone for authentication.
+
+ * The new version, 20.2.1, fixes this issue by reverting the behaviour of
+ the http handler to a way more in line with the implementation in
+ Squid (19.2.3).
+
+ [ Test Plan ]
+
+ * The issue can be reproduced by running a basic S3 workflow with the
+ RADOS gateway and using keystone for authentication.
+
+ * We will verify that the issue is no longer present by running the functional
+ test suite for the Ceph charms, which include appropriate tests for RGW.
+
+ [ Where problems could occur ]
+
+ * The new version reverts part of the behaviour in the HTTP handling of the
+ RADOS gateway. Since this is an integral part of the gateway, any issues
+ that are introduced by these changes should be immediately visible.
+
+ * The new version uses the same implementation as what it was in the previous
+ packages (19.2.3). No issues with RADOS gateway were found with them.
+
+ [ Original Description ]
+
On Ubuntu resolute radosgw from 20.2.0-0ubuntu2 reproducibly segfaults
when an S3 request is authenticated with Keystone EC2 credentials. The
client sees "502 Bad Gateway" and systemd restarts radosgw.
This is the Keystone-backed S3 auth path. Local RGW users can work when
local auth is tried first, but a request using Keystone EC2 credentials
still crashes RGW.
This can be reproduced using the CI bundle with adaptation for resolute
and zaza tests
Stack trace
```
Thread 602 "http_manager" received signal SIGSEGV, Segmentation fault.
0x00005e52632bf3b6 in ceph::async::detail::CompletionImpl<boost::asio::any_io_executor, boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void (boost::system::error_code), void>, void, boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&) ()
Thread 602 (Thread 0x753657aea6c0 (LWP 63226) "http_manager"):
- #0 0x00005e52632bf3b6 in ceph::async::detail::CompletionImpl<boost::asio::any_io_executor, boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void (boost::system::error_code), void>, void, boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&) ()
- #1 0x00005e5263345d18 in RGWHTTPManager::finish_request(rgw_http_req_data*, int, long) ()
- #2 0x00005e5263346c5a in RGWHTTPManager::reqs_thread_entry() ()
- #3 0x00005e52633475e1 in RGWHTTPManager::ReqsThread::entry() ()
- #4 0x000075365eca40da in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:454
- #5 0x000075365ed377ac in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
+ #0 0x00005e52632bf3b6 in ceph::async::detail::CompletionImpl<boost::asio::any_io_executor, boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void (boost::system::error_code), void>, void, boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&) ()
+ #1 0x00005e5263345d18 in RGWHTTPManager::finish_request(rgw_http_req_data*, int, long) ()
+ #2 0x00005e5263346c5a in RGWHTTPManager::reqs_thread_entry() ()
+ #3 0x00005e52633475e1 in RGWHTTPManager::ReqsThread::entry() ()
+ #4 0x000075365eca40da in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:454
+ #5 0x000075365ed377ac in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
```
Juju status
```
# juju status
- Model Controller Cloud/Region Version SLA Timestamp
- zaza-9bef958d671a lxd localhost/localhost 3.6.23 unsupported 11:39:43Z
+ Model Controller Cloud/Region Version SLA Timestamp
+ zaza-9bef958d671a lxd localhost/localhost 3.6.23 unsupported 11:39:43Z
- App Version Status Scale Charm Channel Rev Exposed Message
- ceph-mon 20.2.0 active 3 ceph-mon 0 no Unit is ready and clustered
- ceph-osd 20.2.0 active 3 ceph-osd 15 no Unit is ready (1 OSD)
- ceph-radosgw 20.2.0 active 1 ceph-radosgw 26 no Unit is ready
- keystone 25.0.0 active 1 keystone latest/edge 774 no Application Ready
- mysql 8.0.44-0ubun... active 1 mysql 8.0/stable 444 no
- vault 1.8.8 active 1 vault latest/edge 363 no Unit is ready (active: true, mlock: disabled)
+ App Version Status Scale Charm Channel Rev Exposed Message
+ ceph-mon 20.2.0 active 3 ceph-mon 0 no Unit is ready and clustered
+ ceph-osd 20.2.0 active 3 ceph-osd 15 no Unit is ready (1 OSD)
+ ceph-radosgw 20.2.0 active 1 ceph-radosgw 26 no Unit is ready
+ keystone 25.0.0 active 1 keystone latest/edge 774 no Application Ready
+ mysql 8.0.44-0ubun... active 1 mysql 8.0/stable 444 no
+ vault 1.8.8 active 1 vault latest/edge 363 no Unit is ready (active: true, mlock: disabled)
- Unit Workload Agent Machine Public address Ports Message
- ceph-mon/0* active idle 4 10.33.157.168 Unit is ready and clustered
- ceph-mon/1 active idle 5 10.33.157.25 Unit is ready and clustered
- ceph-mon/2 active idle 6 fd42:34da:d2af:4f10:216:3eff:fea3:1a7b Unit is ready and clustered
- ceph-osd/0* active idle 1 fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 Unit is ready (1 OSD)
- ceph-osd/1 active idle 2 10.33.157.197 Unit is ready (1 OSD)
- ceph-osd/2 active idle 3 10.33.157.131 Unit is ready (1 OSD)
- ceph-radosgw/0* active idle 0 fd42:34da:d2af:4f10:216:3eff:fece:c073 80/tcp Unit is ready
- keystone/0* active idle 8 10.33.157.200 5000/tcp Unit is ready
- mysql/0* active idle 7 10.33.157.76 3306,33060/tcp Primary
- vault/0* active idle 9 fd42:34da:d2af:4f10:216:3eff:fe01:376d 8200/tcp Unit is ready (active: true, mlock: disabled)
+ Unit Workload Agent Machine Public address Ports Message
+ ceph-mon/0* active idle 4 10.33.157.168 Unit is ready and clustered
+ ceph-mon/1 active idle 5 10.33.157.25 Unit is ready and clustered
+ ceph-mon/2 active idle 6 fd42:34da:d2af:4f10:216:3eff:fea3:1a7b Unit is ready and clustered
+ ceph-osd/0* active idle 1 fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 Unit is ready (1 OSD)
+ ceph-osd/1 active idle 2 10.33.157.197 Unit is ready (1 OSD)
+ ceph-osd/2 active idle 3 10.33.157.131 Unit is ready (1 OSD)
+ ceph-radosgw/0* active idle 0 fd42:34da:d2af:4f10:216:3eff:fece:c073 80/tcp Unit is ready
+ keystone/0* active idle 8 10.33.157.200 5000/tcp Unit is ready
+ mysql/0* active idle 7 10.33.157.76 3306,33060/tcp Primary
+ vault/0* active idle 9 fd42:34da:d2af:4f10:216:3eff:fe01:376d 8200/tcp Unit is ready (active: true, mlock: disabled)
- Machine State Address Inst id Base AZ Message
- 0 started fd42:34da:d2af:4f10:216:3eff:fece:c073 juju-a24339-0 ubuntu at 26.04 sunkern Running
- 1 started fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 juju-a24339-1 ubuntu at 26.04 sunkern Running
- 2 started 10.33.157.197 juju-a24339-2 ubuntu at 26.04 sunkern Running
- 3 started 10.33.157.131 juju-a24339-3 ubuntu at 26.04 sunkern Running
- 4 started 10.33.157.168 juju-a24339-4 ubuntu at 26.04 sunkern Running
- 5 started 10.33.157.25 juju-a24339-5 ubuntu at 26.04 sunkern Running
- 6 started fd42:34da:d2af:4f10:216:3eff:fea3:1a7b juju-a24339-6 ubuntu at 26.04 sunkern Running
- 7 started 10.33.157.76 juju-a24339-7 ubuntu at 22.04 sunkern Running
- 8 started 10.33.157.200 juju-a24339-8 ubuntu at 24.04 sunkern Running
- 9 started fd42:34da:d2af:4f10:216:3eff:fe01:376d juju-a24339-9 ubuntu at 24.04 sunkern Running
+ Machine State Address Inst id Base AZ Message
+ 0 started fd42:34da:d2af:4f10:216:3eff:fece:c073 juju-a24339-0 ubuntu at 26.04 sunkern Running
+ 1 started fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 juju-a24339-1 ubuntu at 26.04 sunkern Running
+ 2 started 10.33.157.197 juju-a24339-2 ubuntu at 26.04 sunkern Running
+ 3 started 10.33.157.131 juju-a24339-3 ubuntu at 26.04 sunkern Running
+ 4 started 10.33.157.168 juju-a24339-4 ubuntu at 26.04 sunkern Running
+ 5 started 10.33.157.25 juju-a24339-5 ubuntu at 26.04 sunkern Running
+ 6 started fd42:34da:d2af:4f10:216:3eff:fea3:1a7b juju-a24339-6 ubuntu at 26.04 sunkern Running
+ 7 started 10.33.157.76 juju-a24339-7 ubuntu at 22.04 sunkern Running
+ 8 started 10.33.157.200 juju-a24339-8 ubuntu at 24.04 sunkern Running
+ 9 started fd42:34da:d2af:4f10:216:3eff:fe01:376d juju-a24339-9 ubuntu at 24.04 sunkern Running
```
versions
```
Ubuntu 26.04 LTS (resolute)
- radosgw: 20.2.0-0ubuntu2
- ceph-common: 20.2.0-0ubuntu2
- librados2: 20.2.0-0ubuntu2
+ radosgw: 20.2.0-0ubuntu2
+ ceph-common: 20.2.0-0ubuntu2
+ librados2: 20.2.0-0ubuntu2
radosgw --version:
ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)
```
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2154304
Title:
ceph radosgw segv with keystone auth
Status in ceph package in Ubuntu:
New
Bug description:
[ Impact ]
* The Ceph RADOS gateway is consistently crashing in version 20.2.0,
on Ubuntu resolute, when used keystone for authentication.
* The new version, 20.2.1, fixes this issue by reverting the behaviour of
the http handler to a way more in line with the implementation in
Squid (19.2.3).
[ Test Plan ]
* The issue can be reproduced by running a basic S3 workflow with the
RADOS gateway and using keystone for authentication.
* We will verify that the issue is no longer present by running the functional
test suite for the Ceph charms, which include appropriate tests for RGW.
[ Where problems could occur ]
* The new version reverts part of the behaviour in the HTTP handling of the
RADOS gateway. Since this is an integral part of the gateway, any issues
that are introduced by these changes should be immediately visible.
* The new version uses the same implementation as what it was in the previous
packages (19.2.3). No issues with RADOS gateway were found with them.
[ Original Description ]
On Ubuntu resolute radosgw from 20.2.0-0ubuntu2 reproducibly segfaults
when an S3 request is authenticated with Keystone EC2 credentials. The
client sees "502 Bad Gateway" and systemd restarts radosgw.
This is the Keystone-backed S3 auth path. Local RGW users can work
when local auth is tried first, but a request using Keystone EC2
credentials still crashes RGW.
This can be reproduced using the CI bundle with adaptation for
resolute and zaza tests
Stack trace
```
Thread 602 "http_manager" received signal SIGSEGV, Segmentation fault.
0x00005e52632bf3b6 in ceph::async::detail::CompletionImpl<boost::asio::any_io_executor, boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void (boost::system::error_code), void>, void, boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&) ()
Thread 602 (Thread 0x753657aea6c0 (LWP 63226) "http_manager"):
#0 0x00005e52632bf3b6 in ceph::async::detail::CompletionImpl<boost::asio::any_io_executor, boost::asio::detail::spawn_handler<boost::asio::any_io_executor, void (boost::system::error_code), void>, void, boost::system::error_code>::destroy_post(std::tuple<boost::system::error_code>&&) ()
#1 0x00005e5263345d18 in RGWHTTPManager::finish_request(rgw_http_req_data*, int, long) ()
#2 0x00005e5263346c5a in RGWHTTPManager::reqs_thread_entry() ()
#3 0x00005e52633475e1 in RGWHTTPManager::ReqsThread::entry() ()
#4 0x000075365eca40da in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:454
#5 0x000075365ed377ac in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
```
Juju status
```
# juju status
Model Controller Cloud/Region Version SLA Timestamp
zaza-9bef958d671a lxd localhost/localhost 3.6.23 unsupported 11:39:43Z
App Version Status Scale Charm Channel Rev Exposed Message
ceph-mon 20.2.0 active 3 ceph-mon 0 no Unit is ready and clustered
ceph-osd 20.2.0 active 3 ceph-osd 15 no Unit is ready (1 OSD)
ceph-radosgw 20.2.0 active 1 ceph-radosgw 26 no Unit is ready
keystone 25.0.0 active 1 keystone latest/edge 774 no Application Ready
mysql 8.0.44-0ubun... active 1 mysql 8.0/stable 444 no
vault 1.8.8 active 1 vault latest/edge 363 no Unit is ready (active: true, mlock: disabled)
Unit Workload Agent Machine Public address Ports Message
ceph-mon/0* active idle 4 10.33.157.168 Unit is ready and clustered
ceph-mon/1 active idle 5 10.33.157.25 Unit is ready and clustered
ceph-mon/2 active idle 6 fd42:34da:d2af:4f10:216:3eff:fea3:1a7b Unit is ready and clustered
ceph-osd/0* active idle 1 fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 Unit is ready (1 OSD)
ceph-osd/1 active idle 2 10.33.157.197 Unit is ready (1 OSD)
ceph-osd/2 active idle 3 10.33.157.131 Unit is ready (1 OSD)
ceph-radosgw/0* active idle 0 fd42:34da:d2af:4f10:216:3eff:fece:c073 80/tcp Unit is ready
keystone/0* active idle 8 10.33.157.200 5000/tcp Unit is ready
mysql/0* active idle 7 10.33.157.76 3306,33060/tcp Primary
vault/0* active idle 9 fd42:34da:d2af:4f10:216:3eff:fe01:376d 8200/tcp Unit is ready (active: true, mlock: disabled)
Machine State Address Inst id Base AZ Message
0 started fd42:34da:d2af:4f10:216:3eff:fece:c073 juju-a24339-0 ubuntu at 26.04 sunkern Running
1 started fd42:34da:d2af:4f10:216:3eff:fe5d:8e95 juju-a24339-1 ubuntu at 26.04 sunkern Running
2 started 10.33.157.197 juju-a24339-2 ubuntu at 26.04 sunkern Running
3 started 10.33.157.131 juju-a24339-3 ubuntu at 26.04 sunkern Running
4 started 10.33.157.168 juju-a24339-4 ubuntu at 26.04 sunkern Running
5 started 10.33.157.25 juju-a24339-5 ubuntu at 26.04 sunkern Running
6 started fd42:34da:d2af:4f10:216:3eff:fea3:1a7b juju-a24339-6 ubuntu at 26.04 sunkern Running
7 started 10.33.157.76 juju-a24339-7 ubuntu at 22.04 sunkern Running
8 started 10.33.157.200 juju-a24339-8 ubuntu at 24.04 sunkern Running
9 started fd42:34da:d2af:4f10:216:3eff:fe01:376d juju-a24339-9 ubuntu at 24.04 sunkern Running
```
versions
```
Ubuntu 26.04 LTS (resolute)
radosgw: 20.2.0-0ubuntu2
ceph-common: 20.2.0-0ubuntu2
librados2: 20.2.0-0ubuntu2
radosgw --version:
ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo)
```
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2154304/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list