[Bug 2161211] Re: ceph-osd crashes due to a data race
Ponnuvel Palaniyappan
2161211 at bugs.launchpad.net
Sat Jul 18 20:16:58 UTC 2026
** Description changed:
- ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
- The crash is caused by a data race between
- the timer thread reading ScrubJob::schedule.scheduled_at (under jobs_lock)
- and a PG thread writing to it via ScrubJob::update_schedule() (without
- jobs_lock).
- ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
+ ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
+ The crash is caused by a data race between the timer thread reading ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread writing to it via ScrubJob::update_schedule() (without jobs_lock). ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
The crash is caused by a data race between the timer thread reading
ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread
writing to it via ScrubJob::update_schedule() (without jobs_lock).
# ceph crash info 2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016
{
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x76e77e242520]",
"/usr/bin/ceph-osd(+0x8d198f) [0x5577b75b998f]",
"/usr/bin/ceph-osd(+0x8d1a23) [0x5577b75b9a23]",
"(ScrubQueue::collect_ripe_jobs(std::vector<boost::intrusive_ptr<ScrubQueue::ScrubJob>, std::allocator<boost::intrusive_ptr<ScrubQueue::ScrubJob> > >&, utime_t)+0x1db) [0x5577b75bce9b]",
"(ScrubQueue::select_pg_and_scrub(Scrub::ScrubPreconds&)+0x104) [0x5577b75c2914]",
"(OSD::sched_scrub()+0x1f5) [0x5577b7382ca5]",
"(OSD::tick_without_osd_lock()+0x498) [0x5577b73bc2f8]",
"(Context::complete(int)+0xd) [0x5577b73cba2d]",
"(CommonSafeTimer<std::mutex>::timer_thread()+0x10c) [0x5577b7a6ddcc]",
"(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x5577b7a6ede1]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76e77e294ac3]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x76e77e3268c0]"
],
"ceph_version": "17.2.9",
"crash_id": "2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016",
"entity_name": "osd.197",
"os_id": "22.04",
"os_name": "Ubuntu 22.04.5 LTS",
"os_version": "22.04.5 LTS (Jammy Jellyfish)",
"os_version_id": "22.04",
"process_name": "ceph-osd",
"stack_sig": "8e59dd0b473e52714e12c295fb445f22d04e2f2239c1dabae9f65f7ebd0d8ee1",
"timestamp": "2026-01-24T19:55:40.134588Z",
"utsname_hostname": "dc1cpbaudosd010",
"utsname_machine": "x86_64",
"utsname_release": "6.8.0-85-generic",
"utsname_sysname": "Linux",
"utsname_version": "#85~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 16:18:59 UTC 2"
}
-
- The crash is intermittent and very likely to happen when several PGs are rescheduled simultaneously.
+ The crash is intermittent and very likely to happen when several PGs are
+ rescheduled simultaneously.
** Description changed:
ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
The crash is caused by a data race between the timer thread reading ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread writing to it via ScrubJob::update_schedule() (without jobs_lock). ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
- The crash is caused by a data race between the timer thread reading
- ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread
- writing to it via ScrubJob::update_schedule() (without jobs_lock).
+ The crash appeears to be caused by a data race between the timer thread
+ reading ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG
+ thread writing to it via ScrubJob::update_schedule() (without
+ jobs_lock).
# ceph crash info 2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016
{
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x76e77e242520]",
"/usr/bin/ceph-osd(+0x8d198f) [0x5577b75b998f]",
"/usr/bin/ceph-osd(+0x8d1a23) [0x5577b75b9a23]",
"(ScrubQueue::collect_ripe_jobs(std::vector<boost::intrusive_ptr<ScrubQueue::ScrubJob>, std::allocator<boost::intrusive_ptr<ScrubQueue::ScrubJob> > >&, utime_t)+0x1db) [0x5577b75bce9b]",
"(ScrubQueue::select_pg_and_scrub(Scrub::ScrubPreconds&)+0x104) [0x5577b75c2914]",
"(OSD::sched_scrub()+0x1f5) [0x5577b7382ca5]",
"(OSD::tick_without_osd_lock()+0x498) [0x5577b73bc2f8]",
"(Context::complete(int)+0xd) [0x5577b73cba2d]",
"(CommonSafeTimer<std::mutex>::timer_thread()+0x10c) [0x5577b7a6ddcc]",
"(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x5577b7a6ede1]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76e77e294ac3]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x76e77e3268c0]"
],
"ceph_version": "17.2.9",
"crash_id": "2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016",
"entity_name": "osd.197",
"os_id": "22.04",
"os_name": "Ubuntu 22.04.5 LTS",
"os_version": "22.04.5 LTS (Jammy Jellyfish)",
"os_version_id": "22.04",
"process_name": "ceph-osd",
"stack_sig": "8e59dd0b473e52714e12c295fb445f22d04e2f2239c1dabae9f65f7ebd0d8ee1",
"timestamp": "2026-01-24T19:55:40.134588Z",
"utsname_hostname": "dc1cpbaudosd010",
"utsname_machine": "x86_64",
"utsname_release": "6.8.0-85-generic",
"utsname_sysname": "Linux",
"utsname_version": "#85~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 16:18:59 UTC 2"
}
The crash is intermittent and very likely to happen when several PGs are
rescheduled simultaneously.
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2161211
Title:
ceph-osd crashes due to a data race
Status in ceph package in Ubuntu:
New
Bug description:
ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
The crash is caused by a data race between the timer thread reading ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread writing to it via ScrubJob::update_schedule() (without jobs_lock). ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs().
The crash appeears to be caused by a data race between the timer
thread reading ScrubJob::schedule.scheduled_at (under jobs_lock) and a
PG thread writing to it via ScrubJob::update_schedule() (without
jobs_lock).
# ceph crash info 2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016
{
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x76e77e242520]",
"/usr/bin/ceph-osd(+0x8d198f) [0x5577b75b998f]",
"/usr/bin/ceph-osd(+0x8d1a23) [0x5577b75b9a23]",
"(ScrubQueue::collect_ripe_jobs(std::vector<boost::intrusive_ptr<ScrubQueue::ScrubJob>, std::allocator<boost::intrusive_ptr<ScrubQueue::ScrubJob> > >&, utime_t)+0x1db) [0x5577b75bce9b]",
"(ScrubQueue::select_pg_and_scrub(Scrub::ScrubPreconds&)+0x104) [0x5577b75c2914]",
"(OSD::sched_scrub()+0x1f5) [0x5577b7382ca5]",
"(OSD::tick_without_osd_lock()+0x498) [0x5577b73bc2f8]",
"(Context::complete(int)+0xd) [0x5577b73cba2d]",
"(CommonSafeTimer<std::mutex>::timer_thread()+0x10c) [0x5577b7a6ddcc]",
"(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x5577b7a6ede1]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76e77e294ac3]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x76e77e3268c0]"
],
"ceph_version": "17.2.9",
"crash_id": "2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016",
"entity_name": "osd.197",
"os_id": "22.04",
"os_name": "Ubuntu 22.04.5 LTS",
"os_version": "22.04.5 LTS (Jammy Jellyfish)",
"os_version_id": "22.04",
"process_name": "ceph-osd",
"stack_sig": "8e59dd0b473e52714e12c295fb445f22d04e2f2239c1dabae9f65f7ebd0d8ee1",
"timestamp": "2026-01-24T19:55:40.134588Z",
"utsname_hostname": "dc1cpbaudosd010",
"utsname_machine": "x86_64",
"utsname_release": "6.8.0-85-generic",
"utsname_sysname": "Linux",
"utsname_version": "#85~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 16:18:59 UTC 2"
}
The crash is intermittent and very likely to happen when several PGs
are rescheduled simultaneously.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2161211/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list