[Bug 1828617] Re: Hosts randomly 'losing' disks, breaking ceph-osd service enumeration

[Launchpad Bug Tracker]

This bug was fixed in the package ceph - 13.2.6-0ubuntu0.19.04.4

---------------
ceph (13.2.6-0ubuntu0.19.04.4) disco; urgency=medium

  [ Eric Desrochers ]
  * Ensure that daemons are not automatically restarted during package
    upgrades (LP: #1840347):
    - d/rules: Use "--no-restart-after-upgrade" and "--no-stop-on-upgrade"
      instead of "--no-restart-on-upgrade".
    - d/rules: Drop exclusion for ceph-[osd,mon,mds] for restarts.

  [ James Page ]
  * d/p/ceph-volume-wait-for-lvs.patch: Cherry pick inflight fix to
    ensure that required wal and db devices are present before
    activating OSD's (LP: #1828617).

  [ Jesse Williamson ]
  * d/p/civetweb-755-1.8-somaxconn-configurable*.patch: Backport changes
    to civetweb to allow tuning of SOMAXCONN in Ceph RADOS Gateway
    deployments (LP: #1838109).

 -- James Page <james.page at ubuntu.com>  Fri, 30 Aug 2019 10:10:04 +0100

** Changed in: ceph (Ubuntu Disco)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1828617

Title:
  Hosts randomly 'losing' disks, breaking ceph-osd service enumeration

Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive queens series:
  In Progress
Status in Ubuntu Cloud Archive rocky series:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Committed
Status in Ubuntu Cloud Archive train series:
  In Progress
Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Bionic:
  Fix Committed
Status in ceph source package in Disco:
  Fix Released
Status in ceph source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  For deployments where the bluestore DB and WAL devices are on separate underlying OSD's, its possible on reboot that the LV's configured on these devices have not yet been scanned and detected; the OSD boot process ignores this fact and tries to boot the OSD anyway as soon as the primary LV supporting the OSD is detected, resulting in the OSD crashing as required block device symlinks are not present.

  [Test Case]
  Deploy ceph with bluestore + separate DB and WAL devices.
  Reboot servers
  OSD devices will fail to start after reboot (its a race so not always).

  [Regression Potential]
  Low - the fix has been landed upstream and simple ensures that if a separate LV is expected for the DB and WAL devices for an OSD, the OSD will not try to boot until they are present.

  [Original Bug Report]
  Ubuntu 18.04.2 Ceph deployment.

  Ceph OSD devices utilizing LVM volumes pointing to udev-based physical devices.
  LVM module is supposed to create PVs from devices using the links in /dev/disk/by-dname/ folder that are created by udev.
  However on reboot it happens (not always, rather like race condition) that Ceph services cannot start, and pvdisplay doesn't show any volumes created. The folder /dev/disk/by-dname/ however has all necessary device created by the end of boot process.

  The behaviour can be fixed manually by running "#/sbin/lvm pvscan
  --cache --activate ay /dev/nvme0n1" command for re-activating the LVM
  components and then the services can be started.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1828617/+subscriptions