[Bug 2161211] [NEW] ceph-osd crashes due to a data race

[Ponnuvel Palaniyappan]

Public bug reported:

ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs(). 
The crash is caused by a data race between
the timer thread reading ScrubJob::schedule.scheduled_at (under jobs_lock)
and a PG thread writing to it via ScrubJob::update_schedule() (without
jobs_lock).
ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs(). 

The crash is caused by a data race between the timer thread reading
ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread
writing to it via ScrubJob::update_schedule() (without jobs_lock).

# ceph crash info 2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016
{
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x76e77e242520]",
"/usr/bin/ceph-osd(+0x8d198f) [0x5577b75b998f]",
"/usr/bin/ceph-osd(+0x8d1a23) [0x5577b75b9a23]",
"(ScrubQueue::collect_ripe_jobs(std::vector<boost::intrusive_ptr<ScrubQueue::ScrubJob>, std::allocator<boost::intrusive_ptr<ScrubQueue::ScrubJob> > >&, utime_t)+0x1db) [0x5577b75bce9b]",
"(ScrubQueue::select_pg_and_scrub(Scrub::ScrubPreconds&)+0x104) [0x5577b75c2914]",
"(OSD::sched_scrub()+0x1f5) [0x5577b7382ca5]",
"(OSD::tick_without_osd_lock()+0x498) [0x5577b73bc2f8]",
"(Context::complete(int)+0xd) [0x5577b73cba2d]",
"(CommonSafeTimer<std::mutex>::timer_thread()+0x10c) [0x5577b7a6ddcc]",
"(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x5577b7a6ede1]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76e77e294ac3]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x76e77e3268c0]"
],
"ceph_version": "17.2.9",
"crash_id": "2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016",
"entity_name": "osd.197",
"os_id": "22.04",
"os_name": "Ubuntu 22.04.5 LTS",
"os_version": "22.04.5 LTS (Jammy Jellyfish)",
"os_version_id": "22.04",
"process_name": "ceph-osd",
"stack_sig": "8e59dd0b473e52714e12c295fb445f22d04e2f2239c1dabae9f65f7ebd0d8ee1",
"timestamp": "2026-01-24T19:55:40.134588Z",
"utsname_hostname": "dc1cpbaudosd010",
"utsname_machine": "x86_64",
"utsname_release": "6.8.0-85-generic",
"utsname_sysname": "Linux",
"utsname_version": "#85~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 16:18:59 UTC 2"
}


The crash is intermittent and very likely to happen when several PGs are rescheduled simultaneously.

** Affects: ceph (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/2161211

Title:
  ceph-osd crashes due to a data race

Status in ceph package in Ubuntu:
  New

Bug description:
  ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs(). 
  The crash is caused by a data race between
  the timer thread reading ScrubJob::schedule.scheduled_at (under jobs_lock)
  and a PG thread writing to it via ScrubJob::update_schedule() (without
  jobs_lock).
  ceph-osd processes crash intermittently in ScrubQueue::collect_ripe_jobs(). 

  The crash is caused by a data race between the timer thread reading
  ScrubJob::schedule.scheduled_at (under jobs_lock) and a PG thread
  writing to it via ScrubJob::update_schedule() (without jobs_lock).

  # ceph crash info 2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016
  {
  "backtrace": [
  "/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x76e77e242520]",
  "/usr/bin/ceph-osd(+0x8d198f) [0x5577b75b998f]",
  "/usr/bin/ceph-osd(+0x8d1a23) [0x5577b75b9a23]",
  "(ScrubQueue::collect_ripe_jobs(std::vector<boost::intrusive_ptr<ScrubQueue::ScrubJob>, std::allocator<boost::intrusive_ptr<ScrubQueue::ScrubJob> > >&, utime_t)+0x1db) [0x5577b75bce9b]",
  "(ScrubQueue::select_pg_and_scrub(Scrub::ScrubPreconds&)+0x104) [0x5577b75c2914]",
  "(OSD::sched_scrub()+0x1f5) [0x5577b7382ca5]",
  "(OSD::tick_without_osd_lock()+0x498) [0x5577b73bc2f8]",
  "(Context::complete(int)+0xd) [0x5577b73cba2d]",
  "(CommonSafeTimer<std::mutex>::timer_thread()+0x10c) [0x5577b7a6ddcc]",
  "(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x5577b7a6ede1]",
  "/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x76e77e294ac3]",
  "/lib/x86_64-linux-gnu/libc.so.6(+0x1268c0) [0x76e77e3268c0]"
  ],
  "ceph_version": "17.2.9",
  "crash_id": "2026-01-24T19:55:40.134588Z_0788dbf1-0cbc-4e7a-bb94-455c81a45016",
  "entity_name": "osd.197",
  "os_id": "22.04",
  "os_name": "Ubuntu 22.04.5 LTS",
  "os_version": "22.04.5 LTS (Jammy Jellyfish)",
  "os_version_id": "22.04",
  "process_name": "ceph-osd",
  "stack_sig": "8e59dd0b473e52714e12c295fb445f22d04e2f2239c1dabae9f65f7ebd0d8ee1",
  "timestamp": "2026-01-24T19:55:40.134588Z",
  "utsname_hostname": "dc1cpbaudosd010",
  "utsname_machine": "x86_64",
  "utsname_release": "6.8.0-85-generic",
  "utsname_sysname": "Linux",
  "utsname_version": "#85~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 16:18:59 UTC 2"
  }

  
  The crash is intermittent and very likely to happen when several PGs are rescheduled simultaneously.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2161211/+subscriptions