[Bug 1805145] Re: radosgw crashed with SIGABRT in __GI_raise() when uploading small file via Horizon/CLI
James Page
james.page at ubuntu.com
Tue Nov 27 09:50:57 UTC 2018
Also reported upstream: https://tracker.ceph.com/issues/37386
** Description changed:
+ [Impact]
+ radosgw is non functional - files cannot be uploaded.
+
+ [Test Case]
+ Deploy ceph + radosgw
+ Upload file using swift cli tooling
+ radosgw daemon SIGABRT's.
+
+ [Regression Potential]
+ The cause of this bug is a distro patch for 32bit compatibility which mis-aligns the type for the retrieval of a configuration option with the underlying type of the option - this results in a SIGABRT.
+
+ The update to the distro patch realigns the types and aligns with a
+ broader patch in upstream ceph in master branch development.
+
+ [Original Bug Report]
Hi,
Juju deployed OpenStack using a modified openstack-core bundle - Rocky
on bionic. All components (except neutron-gateway) are deployed in LXD
containers.
Horizon/CLI can list and create new containers, however, if a small file
(in this case a cloud.init file of 82 bytes) is uploaded, the radosgw
process crashes with attached crash file.
NB this is running within GCP
Please advise if any further information required. Here's the snippet
out of the end of radosgw.log
d9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e73f9740 con 0
- -27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.179:0/1413084365 >> 252.0.80.134:6800/7301 conn(0x5587e6c51800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.0 seq 10832 0x5587e73f9740 osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
- -26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.179:0/1413084365 <== osd.0 252.0.80.134:6800/7301 10832 ==== osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (2451902153 0 0) 0x5587e73f9740 con 0x5587e6c51800
- -25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:normalizing buckets and tenants
- -24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init permissions
- -23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:recalculating target
- -22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:reading permissions
- -21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init op
- -20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op mask
- -19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op permissions
- -18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess
- -17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxx_mgmt, perm_mask=15, is_admin=0) mask=50
- -16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f
- -15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15
- -14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f
- -13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found
- -12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50
- -11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
- -10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50
- -9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
- -8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxxx_mgmt, perm_mask=15, is_admin=0), owner=ad9fa6cacbb94b96b4941ec2af38a49f, perm=2
- -7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op params
- -6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:pre-executing
- -5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:executing
- -4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.179:0/1413084365 --> 252.0.90.230:6800/17596 -- osd_op(unknown.0.0:132198 6.0 6:21b9d9f7:::.dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e745a9c0 con 0
- -3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.179:0/1413084365 >> 252.0.90.230:6800/17596 conn(0x5587e6c50300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10 cs=1 l=1). rx osd.1 seq 54313 0x5587e745a9c0 osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8
- -2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.179:0/1413084365 <== osd.1 252.0.90.230:6800/17596 54313 ==== osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 ==== 192+0+71 (1074935246 0 1943469035) 0x5587e745a9c0 con 0x5587e6c50300
- -1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_auth_completion
- 0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) **
- in thread 7f87b633a700 thread_name:civetweb-worker
+ -27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.179:0/1413084365 >> 252.0.80.134:6800/7301 conn(0x5587e6c51800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.0 seq 10832 0x5587e73f9740 osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
+ -26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.179:0/1413084365 <== osd.0 252.0.80.134:6800/7301 10832 ==== osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (2451902153 0 0) 0x5587e73f9740 con 0x5587e6c51800
+ -25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:normalizing buckets and tenants
+ -24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init permissions
+ -23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:recalculating target
+ -22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:reading permissions
+ -21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init op
+ -20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op mask
+ -19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op permissions
+ -18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess
+ -17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxx_mgmt, perm_mask=15, is_admin=0) mask=50
+ -16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f
+ -15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15
+ -14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f
+ -13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found
+ -12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50
+ -11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
+ -10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50
+ -9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
+ -8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxxx_mgmt, perm_mask=15, is_admin=0), owner=ad9fa6cacbb94b96b4941ec2af38a49f, perm=2
+ -7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op params
+ -6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:pre-executing
+ -5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:executing
+ -4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.179:0/1413084365 --> 252.0.90.230:6800/17596 -- osd_op(unknown.0.0:132198 6.0 6:21b9d9f7:::.dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e745a9c0 con 0
+ -3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.179:0/1413084365 >> 252.0.90.230:6800/17596 conn(0x5587e6c50300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10 cs=1 l=1). rx osd.1 seq 54313 0x5587e745a9c0 osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8
+ -2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.179:0/1413084365 <== osd.1 252.0.90.230:6800/17596 54313 ==== osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 ==== 192+0+71 (1074935246 0 1943469035) 0x5587e745a9c0 con 0x5587e6c50300
+ -1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_auth_completion
+ 0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) **
+ in thread 7f87b633a700 thread_name:civetweb-worker
- ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)
- 1: (()+0x293e70) [0x5587e4514e70]
- 2: (()+0x12890) [0x7f87eb99d890]
- 3: (gsignal()+0xc7) [0x7f87e0e96e97]
- 4: (abort()+0x141) [0x7f87e0e98801]
- 5: (()+0x8c8b7) [0x7f87e14ed8b7]
- 6: (()+0x92a06) [0x7f87e14f3a06]
- 7: (()+0x92a41) [0x7f87e14f3a41]
- 8: (()+0x92c74) [0x7f87e14f3c74]
- 9: (()+0x35e30d) [0x5587e45df30d]
- 10: (Option::size_t const md_config_t::get_val<Option::size_t>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x53) [0x5587e461fab3]
- 11: (()+0x37df4f) [0x5587e45fef4f]
- 12: (RGWPutObj::execute()+0x1adb) [0x5587e4600efb]
- 13: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x5587e4620cfd]
- 14: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0x15f0) [0x5587e46223f0]
- 15: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5587e44976e1]
- 16: (()+0x2851ee) [0x5587e45061ee]
- 17: (()+0x286e78) [0x5587e4507e78]
- 18: (()+0x287308) [0x5587e4508308]
- 19: (()+0x76db) [0x7f87eb9926db]
- 20: (clone()+0x3f) [0x7f87e0f7988f]
- NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
+ ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)
+ 1: (()+0x293e70) [0x5587e4514e70]
+ 2: (()+0x12890) [0x7f87eb99d890]
+ 3: (gsignal()+0xc7) [0x7f87e0e96e97]
+ 4: (abort()+0x141) [0x7f87e0e98801]
+ 5: (()+0x8c8b7) [0x7f87e14ed8b7]
+ 6: (()+0x92a06) [0x7f87e14f3a06]
+ 7: (()+0x92a41) [0x7f87e14f3a41]
+ 8: (()+0x92c74) [0x7f87e14f3c74]
+ 9: (()+0x35e30d) [0x5587e45df30d]
+ 10: (Option::size_t const md_config_t::get_val<Option::size_t>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x53) [0x5587e461fab3]
+ 11: (()+0x37df4f) [0x5587e45fef4f]
+ 12: (RGWPutObj::execute()+0x1adb) [0x5587e4600efb]
+ 13: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x5587e4620cfd]
+ 14: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0x15f0) [0x5587e46223f0]
+ 15: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5587e44976e1]
+ 16: (()+0x2851ee) [0x5587e45061ee]
+ 17: (()+0x286e78) [0x5587e4507e78]
+ 18: (()+0x287308) [0x5587e4508308]
+ 19: (()+0x76db) [0x7f87eb9926db]
+ 20: (clone()+0x3f) [0x7f87e0f7988f]
+ NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
- 0/ 5 none
- 0/ 1 lockdep
- 0/ 1 context
- 1/ 1 crush
- 1/ 5 mds
- 1/ 5 mds_balancer
- 1/ 5 mds_locker
- 1/ 5 mds_log
- 1/ 5 mds_log_expire
- 1/ 5 mds_migrator
- 0/ 1 buffer
- 0/ 1 timer
- 0/ 1 filer
- 0/ 1 striper
- 0/ 1 objecter
- 0/ 5 rados
- 0/ 5 rbd
- 0/ 5 rbd_mirror
- 0/ 5 rbd_replay
- 0/ 5 journaler
- 0/ 5 objectcacher
- 0/ 5 client
- 1/ 5 osd
- 0/ 5 optracker
- 0/ 5 objclass
- 1/ 3 filestore
- 1/ 3 journal
- 0/ 5 ms
- 1/ 5 mon
- 0/10 monc
- 1/ 5 paxos
- 0/ 5 tp
- 1/ 5 auth
- 1/ 5 crypto
- 1/ 1 finisher
- 1/ 1 reserver
- 1/ 5 heartbeatmap
- 1/ 5 perfcounter
- 1/ 5 rgw
- 1/ 5 rgw_sync
- 1/10 civetweb
- 1/ 5 javaclient
- 1/ 5 asok
- 1/ 1 throttle
- 0/ 0 refs
- 1/ 5 xio
- 1/ 5 compressor
- 1/ 5 bluestore
- 1/ 5 bluefs
- 1/ 3 bdev
- 1/ 5 kstore
- 4/ 5 rocksdb
- 4/ 5 leveldb
- 4/ 5 memdb
- 1/ 5 kinetic
- 1/ 5 fuse
- 1/ 5 mgr
- 1/ 5 mgrc
- 1/ 5 dpdk
- 1/ 5 eventtrace
- -2/-2 (syslog threshold)
- -1/-1 (stderr threshold)
- max_recent 10000
- max_new 1000
- log_file /var/log/ceph/radosgw.log
+ 0/ 5 none
+ 0/ 1 lockdep
+ 0/ 1 context
+ 1/ 1 crush
+ 1/ 5 mds
+ 1/ 5 mds_balancer
+ 1/ 5 mds_locker
+ 1/ 5 mds_log
+ 1/ 5 mds_log_expire
+ 1/ 5 mds_migrator
+ 0/ 1 buffer
+ 0/ 1 timer
+ 0/ 1 filer
+ 0/ 1 striper
+ 0/ 1 objecter
+ 0/ 5 rados
+ 0/ 5 rbd
+ 0/ 5 rbd_mirror
+ 0/ 5 rbd_replay
+ 0/ 5 journaler
+ 0/ 5 objectcacher
+ 0/ 5 client
+ 1/ 5 osd
+ 0/ 5 optracker
+ 0/ 5 objclass
+ 1/ 3 filestore
+ 1/ 3 journal
+ 0/ 5 ms
+ 1/ 5 mon
+ 0/10 monc
+ 1/ 5 paxos
+ 0/ 5 tp
+ 1/ 5 auth
+ 1/ 5 crypto
+ 1/ 1 finisher
+ 1/ 1 reserver
+ 1/ 5 heartbeatmap
+ 1/ 5 perfcounter
+ 1/ 5 rgw
+ 1/ 5 rgw_sync
+ 1/10 civetweb
+ 1/ 5 javaclient
+ 1/ 5 asok
+ 1/ 1 throttle
+ 0/ 0 refs
+ 1/ 5 xio
+ 1/ 5 compressor
+ 1/ 5 bluestore
+ 1/ 5 bluefs
+ 1/ 3 bdev
+ 1/ 5 kstore
+ 4/ 5 rocksdb
+ 4/ 5 leveldb
+ 4/ 5 memdb
+ 1/ 5 kinetic
+ 1/ 5 fuse
+ 1/ 5 mgr
+ 1/ 5 mgrc
+ 1/ 5 dpdk
+ 1/ 5 eventtrace
+ -2/-2 (syslog threshold)
+ -1/-1 (stderr threshold)
+ max_recent 10000
+ max_new 1000
+ log_file /var/log/ceph/radosgw.log
--- end dump of recent events ---
ProblemType: Crash
DistroRelease: Ubuntu 18.04
Package: radosgw 13.2.1+dfsg1-0ubuntu2~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-1024.25-gcp 4.15.18
Uname: Linux 4.15.0-1024-gcp x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CrashDB:
- {
- "impl": "launchpad",
- "project": "cloud-archive",
- "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
- }
+ {
+ "impl": "launchpad",
+ "project": "cloud-archive",
+ "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
+ }
Date: Mon Nov 26 12:17:31 2018
ExecutablePath: /usr/bin/radosgw
ProcCmdline: /usr/bin/radosgw -n client.radosgw.gateway
ProcEnviron:
- PATH=(custom, no user)
- LANG=C.UTF-8
+ PATH=(custom, no user)
+ LANG=C.UTF-8
Signal: 6
SourcePackage: ceph
StacktraceTop:
- __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
- __GI_abort () at abort.c:79
- ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
- ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
- std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
+ __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
+ __GI_abort () at abort.c:79
+ ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
+ ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
+ std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Title: radosgw crashed with SIGABRT in __GI_raise()
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1805145
Title:
radosgw crashed with SIGABRT in __GI_raise() when uploading small file
via Horizon/CLI
Status in Ubuntu Cloud Archive:
In Progress
Status in Ubuntu Cloud Archive rocky series:
In Progress
Status in Ubuntu Cloud Archive stein series:
In Progress
Status in ceph package in Ubuntu:
In Progress
Status in ceph source package in Cosmic:
In Progress
Status in ceph source package in Disco:
In Progress
Bug description:
[Impact]
radosgw is non functional - files cannot be uploaded.
[Test Case]
Deploy ceph + radosgw
Upload file using swift cli tooling
radosgw daemon SIGABRT's.
[Regression Potential]
The cause of this bug is a distro patch for 32bit compatibility which mis-aligns the type for the retrieval of a configuration option with the underlying type of the option - this results in a SIGABRT.
The update to the distro patch realigns the types and aligns with a
broader patch in upstream ceph in master branch development.
[Original Bug Report]
Hi,
Juju deployed OpenStack using a modified openstack-core bundle - Rocky
on bionic. All components (except neutron-gateway) are deployed in
LXD containers.
Horizon/CLI can list and create new containers, however, if a small
file (in this case a cloud.init file of 82 bytes) is uploaded, the
radosgw process crashes with attached crash file.
NB this is running within GCP
Please advise if any further information required. Here's the snippet
out of the end of radosgw.log
d9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e73f9740 con 0
-27> 2018-11-26 12:17:31.262 7f87d596b700 5 -- 252.0.89.179:0/1413084365 >> 252.0.80.134:6800/7301 conn(0x5587e6c51800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.0 seq 10832 0x5587e73f9740 osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
-26> 2018-11-26 12:17:31.262 7f87d596b700 1 -- 252.0.89.179:0/1413084365 <== osd.0 252.0.80.134:6800/7301 10832 ==== osd_op_reply(132197 ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (2451902153 0 0) 0x5587e73f9740 con 0x5587e6c51800
-25> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001665:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:normalizing buckets and tenants
-24> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001701:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init permissions
-23> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001755:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:recalculating target
-22> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001779:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:reading permissions
-21> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001786:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:init op
-20> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001788:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op mask
-19> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001789:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op permissions
-18> 2018-11-26 12:17:31.262 7f87b633a700 5 content type wasn't provided, trying to guess
-17> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxx_mgmt, perm_mask=15, is_admin=0) mask=50
-16> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f
-15> 2018-11-26 12:17:31.262 7f87b633a700 5 Found permission: 15
-14> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for uid=ad9fa6cacbb94b96b4941ec2af38a49f$ad9fa6cacbb94b96b4941ec2af38a49f
-13> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for user not found
-12> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=1 mask=50
-11> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
-10> 2018-11-26 12:17:31.262 7f87b633a700 5 Searching permissions for group=2 mask=50
-9> 2018-11-26 12:17:31.262 7f87b633a700 5 Permissions for group not found
-8> 2018-11-26 12:17:31.262 7f87b633a700 5 -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::RemoteApplier(acct_user=ad9fa6cacbb94b96b4941ec2af38a49f, acct_name=xxxxx_mgmt, perm_mask=15, is_admin=0), owner=ad9fa6cacbb94b96b4941ec2af38a49f, perm=2
-7> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001818:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:verifying op params
-6> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001820:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:pre-executing
-5> 2018-11-26 12:17:31.262 7f87b633a700 2 req 23:0.001821:swift:PUT /swift/v1/test/cloudinit.txt:put_obj:executing
-4> 2018-11-26 12:17:31.262 7f87b633a700 1 -- 252.0.89.179:0/1413084365 --> 252.0.90.230:6800/17596 -- osd_op(unknown.0.0:132198 6.0 6:21b9d9f7:::.dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e80) v8 -- 0x5587e745a9c0 con 0
-3> 2018-11-26 12:17:31.266 7f87d616c700 5 -- 252.0.89.179:0/1413084365 >> 252.0.90.230:6800/17596 conn(0x5587e6c50300 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=10 cs=1 l=1). rx osd.1 seq 54313 0x5587e745a9c0 osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8
-2> 2018-11-26 12:17:31.266 7f87d616c700 1 -- 252.0.89.179:0/1413084365 <== osd.1 252.0.90.230:6800/17596 54313 ==== osd_op_reply(132198 .dir.c9c6424e-a4bd-44de-b972-6cd4539a0a61.5322.1 [call] v0'0 uv1 ondisk = 0) v8 ==== 192+0+71 (1074935246 0 1943469035) 0x5587e745a9c0 con 0x5587e6c50300
-1> 2018-11-26 12:17:31.266 7f87b633a700 5 NOTICE: call to do_aws4_auth_completion
0> 2018-11-26 12:17:31.266 7f87b633a700 -1 *** Caught signal (Aborted) **
in thread 7f87b633a700 thread_name:civetweb-worker
ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)
1: (()+0x293e70) [0x5587e4514e70]
2: (()+0x12890) [0x7f87eb99d890]
3: (gsignal()+0xc7) [0x7f87e0e96e97]
4: (abort()+0x141) [0x7f87e0e98801]
5: (()+0x8c8b7) [0x7f87e14ed8b7]
6: (()+0x92a06) [0x7f87e14f3a06]
7: (()+0x92a41) [0x7f87e14f3a41]
8: (()+0x92c74) [0x7f87e14f3c74]
9: (()+0x35e30d) [0x5587e45df30d]
10: (Option::size_t const md_config_t::get_val<Option::size_t>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x53) [0x5587e461fab3]
11: (()+0x37df4f) [0x5587e45fef4f]
12: (RGWPutObj::execute()+0x1adb) [0x5587e4600efb]
13: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x5587e4620cfd]
14: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0x15f0) [0x5587e46223f0]
15: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5587e44976e1]
16: (()+0x2851ee) [0x5587e45061ee]
17: (()+0x286e78) [0x5587e4507e78]
18: (()+0x287308) [0x5587e4508308]
19: (()+0x76db) [0x7f87eb9926db]
20: (clone()+0x3f) [0x7f87e0f7988f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/radosgw.log
--- end dump of recent events ---
ProblemType: Crash
DistroRelease: Ubuntu 18.04
Package: radosgw 13.2.1+dfsg1-0ubuntu2~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-1024.25-gcp 4.15.18
Uname: Linux 4.15.0-1024-gcp x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CrashDB:
{
"impl": "launchpad",
"project": "cloud-archive",
"bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
}
Date: Mon Nov 26 12:17:31 2018
ExecutablePath: /usr/bin/radosgw
ProcCmdline: /usr/bin/radosgw -n client.radosgw.gateway
ProcEnviron:
PATH=(custom, no user)
LANG=C.UTF-8
Signal: 6
SourcePackage: ceph
StacktraceTop:
__GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
__GI_abort () at abort.c:79
?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Title: radosgw crashed with SIGABRT in __GI_raise()
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1805145/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list