[Bug 1783542] [NEW] OSD creation on bcache device not working

[Gábor Mészáros]

Public bug reported:

Using the ceph-osd charm on xenial hwe.

The charm successfully calls out for the regular ceph-disk prepare
command, but that fails since it thinks of that device being a partition
and doesn't create partition on top of bcache.

The log ends up with failing:
2018-07-24 14:41:40 INFO juju-log osdize cmd: ['ceph-disk', 'prepare', '--fs-type', 'xfs', '/dev/bcache4', '/dev/sdd']
2018-07-24 14:41:42 DEBUG add-disk set_data_partition: incorrect partition UUID: None, expected ['4fbd7e29-9d25-41b8-afd0-5ec00ceff05d', '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', '4fbd7e29-8ae0-4982-bf9d-5a8d867af560', '4fbd7e29-9d25-41b8-afd0-35865ceff05d']

Full log:
https://pastebin.canonical.com/p/8JMM9JbhxZ/

After inspecting the ceph-disk source, I've found out where it is failing:
ceph/xenial-updates,now 10.2.9-0ubuntu0.16.04.1 amd64 [installed]
/usr/lib/python2.7/dist-packages/ceph_disk/main.py:#763

def is_partition(dev):
    """
    Check whether a given device path is a partition or a full disk.
    """
    if is_mpath(dev):
        return is_partition_mpath(dev)

    dev = os.path.realpath(dev)
    st = os.lstat(dev)
    if not stat.S_ISBLK(st.st_mode):
        raise Error('not a block device', dev)

    name = get_dev_name(dev)
    if is_bcache(name):
        return True
    if os.path.exists(os.path.join('/sys/block', name)):
        return False

If I remove the is_bcache check, it tries to create the partition on top
and succeeds with in only with running on xenial-hwe kernel
(4.13.0-45-generic at the moment).

However patches that support partitioning of bcache devices are not
available on the mainline kernel, so I suspect it would fail when not
having our patches applied. [0]

Note that this is probably related to the LP#1667078 fix [1] (bcache device numbers increase by 16) and 
https://launchpadlibrarian.net/309401983/0001-bcache-Fix-bcache-device-names.patch

Also note that this issue is not related to LP#1729145 or LP#1728742 as
I'm already running on the fixed kernel and the sympthoms are different.
[2]. Also MAAS is up to date. [3]

I assume this patch has been accepted by Canonical (not mainline ceph)
and causing us the issue [4].

[0]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b8c0d911ac5285e6be8967713271a51bdc5a936a
[1]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1667078
[2]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729145
[3]: https://bugs.launchpad.net/curtin/+bug/1728742
[4]: http://tracker.ceph.com/issues/13278

** Affects: ceph (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: 4010

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1783542

Title:
  OSD creation on bcache device not working

Status in ceph package in Ubuntu:
  New

Bug description:
  Using the ceph-osd charm on xenial hwe.

  The charm successfully calls out for the regular ceph-disk prepare
  command, but that fails since it thinks of that device being a
  partition and doesn't create partition on top of bcache.

  The log ends up with failing:
  2018-07-24 14:41:40 INFO juju-log osdize cmd: ['ceph-disk', 'prepare', '--fs-type', 'xfs', '/dev/bcache4', '/dev/sdd']
  2018-07-24 14:41:42 DEBUG add-disk set_data_partition: incorrect partition UUID: None, expected ['4fbd7e29-9d25-41b8-afd0-5ec00ceff05d', '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', '4fbd7e29-8ae0-4982-bf9d-5a8d867af560', '4fbd7e29-9d25-41b8-afd0-35865ceff05d']

  Full log:
  https://pastebin.canonical.com/p/8JMM9JbhxZ/

  After inspecting the ceph-disk source, I've found out where it is failing:
  ceph/xenial-updates,now 10.2.9-0ubuntu0.16.04.1 amd64 [installed]
  /usr/lib/python2.7/dist-packages/ceph_disk/main.py:#763

  def is_partition(dev):
      """
      Check whether a given device path is a partition or a full disk.
      """
      if is_mpath(dev):
          return is_partition_mpath(dev)

      dev = os.path.realpath(dev)
      st = os.lstat(dev)
      if not stat.S_ISBLK(st.st_mode):
          raise Error('not a block device', dev)

      name = get_dev_name(dev)
      if is_bcache(name):
          return True
      if os.path.exists(os.path.join('/sys/block', name)):
          return False

  If I remove the is_bcache check, it tries to create the partition on
  top and succeeds with in only with running on xenial-hwe kernel
  (4.13.0-45-generic at the moment).

  However patches that support partitioning of bcache devices are not
  available on the mainline kernel, so I suspect it would fail when not
  having our patches applied. [0]

  Note that this is probably related to the LP#1667078 fix [1] (bcache device numbers increase by 16) and 
  https://launchpadlibrarian.net/309401983/0001-bcache-Fix-bcache-device-names.patch

  Also note that this issue is not related to LP#1729145 or LP#1728742
  as I'm already running on the fixed kernel and the sympthoms are
  different. [2]. Also MAAS is up to date. [3]

  I assume this patch has been accepted by Canonical (not mainline ceph)
  and causing us the issue [4].

  [0]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b8c0d911ac5285e6be8967713271a51bdc5a936a
  [1]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1667078
  [2]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729145
  [3]: https://bugs.launchpad.net/curtin/+bug/1728742
  [4]: http://tracker.ceph.com/issues/13278

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1783542/+subscriptions