[Bug 1767087] Re: ceph-volume: block device permissions sometimes not set on initial activate call

[James Page]

This is a bit of a race between ceph-volume and udev - if I force the
perms to be correct by starting the ceph-volume systemd unit for an OSD:

$ getfacl /dev/ceph-4a94b8ca-bc68-4ac6-a5ec-dc10549af193/osd-block-4a94b8ca-bc68-4ac6-a5ec-dc10549af193
# file: dev/ceph-4a94b8ca-bc68-4ac6-a5ec-dc10549af193/osd-block-4a94b8ca-bc68-4ac6-a5ec-dc10549af193
# owner: ceph
# group: ceph
user::rw-
group::rw-
other::---

then re-trigger udev events for block devices:

$ udevadm trigger --subsystem-match=block --action=add
$ getfacl /dev/ceph-4a94b8ca-bc68-4ac6-a5ec-dc10549af193/osd-block-4a94b8ca-bc68-4ac6-a5ec-dc10549af193
# file: dev/ceph-4a94b8ca-bc68-4ac6-a5ec-dc10549af193/osd-block-4a94b8ca-bc68-4ac6-a5ec-dc10549af193
# owner: root
# group: disk
user::rw-
group::rw-
other::---

you can see the permissions revert back to root/disk from ceph/ceph.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1767087

Title:
  ceph-volume: block device permissions sometimes not set on initial
  activate call

Status in ceph package in Ubuntu:
  Confirmed

Bug description:
  When using bluestore or filestore with a separate journal device, we
  often see that the ceph-osd daemon crashes on activation with a
  'permission denied' error on the link to the block device supporting
  the particular component of the OSD (bluestore block/wal/db or journal
  device for filestore).

  chown -R ceph:ceph /dev/dm-X fixes the issue and the osd can then be
  started; I've also not seen this issue occur during a normal reboot
  (as the permissions are re-asserted by ceph-volume as OSD's are
  started up).

  Hunch is something udev-y is racing with ceph-volume on the initial
  activiate call but not tracked it down yet.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1767087/+subscriptions