[Bug 1891567] Re: [SRU] ceph_osd crash in _committed_osd_maps when failed to encode first inc map

Robie Basak 1891567 at bugs.launchpad.net
Thu Aug 20 13:54:51 UTC 2020


Hello Dan, or anyone else affected,

Accepted ceph into focal-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/ceph/15.2.3-0ubuntu0.20.04.2 in a
few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
focal to verification-done-focal. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-focal. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: ceph (Ubuntu Focal)
       Status: Triaged => Fix Committed

** Tags added: verification-needed verification-needed-focal

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1891567

Title:
  [SRU] ceph_osd crash in _committed_osd_maps when failed to encode
  first inc map

Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive ussuri series:
  Triaged
Status in Ubuntu Cloud Archive victoria series:
  Triaged
Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Focal:
  Fix Committed
Status in ceph source package in Groovy:
  Fix Released

Bug description:
  [Impact]
  Upstream tracker: issue#46443 [0].

  The ceph-osd service can crash when processing osd map updates.

  When the osd encounters a CRC error while processing an incremental
  map update, it will request a full map update from its peers. In this
  code path, an uninitialized variable was recently introduced and that
  will get de-referenced causing a crash.

  The uninitialized variable was introduced in nautilus 14.2.10, and
  octopus 15.2.1.

  [Test Case]
  # Inject osd_inject_bad_map_crc_probability = 1
  sudo ceph daemon osd.{id} config set osd_inject_bad_map_crc_probability 1

  # Trigger some osd map updates by restarting a different osd
  sudo systemctl restart osd@{diff-id}

  [Regression Potential]
  The code has been updated to leave handle_osd_maps() early if a CRC error is encountered, therefore preventing the map commit if the failure is encountered while processing an incremental map update. This will make the full map update take longer but should prevent the crash that resulted in this bug. Additionally, _committed_osd_maps() is now coded to assert if first <= last, but it is assumed that code should never be reached.

  [Other Info]
  Upstream has released a fix for this issue in Nautilus 14.2.11. The SRU for this point release is being tracked by LP: #1891077

  Upstream has merged a fix for this issue in Octopus [1], but there is
  no current release target. The ceph packages in focal, groovy, and the
  ussuri cloud archive are exposed to this critical regression.

  [0] https://tracker.ceph.com/issues/46443
  [1] https://github.com/ceph/ceph/pull/36340

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1891567/+subscriptions



More information about the Ubuntu-openstack-bugs mailing list