[Bug 1891567] [NEW] [SRU] ceph_osd crash in _committed_osd_maps when failed to encode first inc map

[Dan Hill]

Public bug reported:

[Impact]
Upstream tracker: issue#46443 [0].

The ceph-osd service can crash when processing osd map updates.

When the osd encounters a CRC error while processing an incremental map
update, it will request a full map update from its peers. In this code
path, an uninitialized variable was recently introduced and that will
get de-referenced causing a crash.

The uninitialized variable was introduced in nautilus 14.2.10, and
octopus 15.2.1.

[Test Case]
# Inject osd_inject_bad_map_crc_probability = 1
sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1

# Trigger some osd map updates by restarting a different osd
sudo systemctl restart osd@{diff-id}

[Other Info]
Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077

Upstream has merged a fix for this issue in Octopus [1], but there is no
current release target. The ceph packages in focal, groovy, and the
ussuri cloud archive are exposed to this critical regression.

[0] https://tracker.ceph.com/issues/46443
[1] https://github.com/ceph/ceph/pull/36340

** Affects: ceph (Ubuntu)
     Importance: Undecided
         Status: New

** Affects: ceph (Ubuntu Focal)
     Importance: Undecided
         Status: New

** Affects: ceph (Ubuntu Groovy)
     Importance: Undecided
         Status: New


** Tags: seg sts

** Description changed:

  [Impact]
  Upstream tracker: issue#46443 [0].
  
  The ceph-osd service can crash when processing osd map updates.
  
  When the osd encounters a CRC error while processing an incremental map
  update, it will request a full map update from its peers. In this code
  path, an uninitialized variable was recently introduced and that will
  get de-referenced causing a crash.
  
  The uninitialized variable was introduced in nautilus 14.2.10, and
  octopus 15.2.1.
  
  [Test Case]
  # Inject osd_inject_bad_map_crc_probability = 1
  sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1
  
  # Trigger some osd map updates by restarting a different osd
  sudo systemctl restart osd@{diff-id}
  
  [Other Info]
  Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077
  
- Upstream has merged a fix for this issue in Octopus, but there is no
+ Upstream has merged a fix for this issue in Octopus [1], but there is no
  current release target. The ceph packages in focal, groovy, and the
  ussuri cloud archive are exposed to this critical regression.
  
  [0] https://tracker.ceph.com/issues/46443
  [1] https://github.com/ceph/ceph/pull/36340

** Also affects: ceph (Ubuntu Groovy)
   Importance: Undecided
       Status: New

** Also affects: ceph (Ubuntu Focal)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1891567

Title:
  [SRU] ceph_osd crash in _committed_osd_maps when failed to encode
  first inc map

Status in ceph package in Ubuntu:
  New
Status in ceph source package in Focal:
  New
Status in ceph source package in Groovy:
  New

Bug description:
  [Impact]
  Upstream tracker: issue#46443 [0].

  The ceph-osd service can crash when processing osd map updates.

  When the osd encounters a CRC error while processing an incremental
  map update, it will request a full map update from its peers. In this
  code path, an uninitialized variable was recently introduced and that
  will get de-referenced causing a crash.

  The uninitialized variable was introduced in nautilus 14.2.10, and
  octopus 15.2.1.

  [Test Case]
  # Inject osd_inject_bad_map_crc_probability = 1
  sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1

  # Trigger some osd map updates by restarting a different osd
  sudo systemctl restart osd@{diff-id}

  [Other Info]
  Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077

  Upstream has merged a fix for this issue in Octopus [1], but there is
  no current release target. The ceph packages in focal, groovy, and the
  ussuri cloud archive are exposed to this critical regression.

  [0] https://tracker.ceph.com/issues/46443
  [1] https://github.com/ceph/ceph/pull/36340

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1891567/+subscriptions