[Bug 1557158] Re: Partition created but device doesn't exist (yet?)

[James Page]

** Changed in: ceph (Ubuntu)
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1557158

Title:
  Partition created but device doesn't exist (yet?)

Status in ceph package in Ubuntu:
  Triaged
Status in ceph-osd package in Juju Charms Collection:
  Invalid

Bug description:
  cs:~openstack-charmers-next/trusty/ceph-osd-81

  I wonder if there is a race between partitioning a disk and expecting the corresponding devices to be created. Case in point:
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:10 INFO unit.ceph-osd/4.juju-log server.go:268 mon:72: osdize cmd: ['ceph-disk', 'prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Creating new GPT entries.
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 GPT data structures destroyed! You may now partition the disk using fdisk or
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:14 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 other utilities.
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:16 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Creating new GPT entries.
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:16 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:18 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 The operation has completed successfully.
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 mkfs.xfs: cannot open /dev/sdb1: No such device or address
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 ceph-disk: Error: Command '['/sbin/mkfs', '-t', 'xfs', '-f', '-i', 'size=2048', '--', '/dev/sdb1']' returned non-zero exit status 1
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 ERROR unit.ceph-osd/4.juju-log server.go:268 mon:72: Unable to initialize device: /dev/sdb
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 Traceback (most recent call last):
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 312, in <module>
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40     hooks.execute(sys.argv)
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40     self._hooks[hook_name]()
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 254, in mon_relation
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40     prepare_disks_and_activate()
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/mon-relation-changed", line 185, in prepare_disks_and_activate
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40     config('osd-encrypt'))
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/ceph.py", line 435, in osdize
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40     reformat_osd, ignore_errors, encrypt)
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-osd-4/charm/hooks/ceph.py", line 490, in osdize_dev
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40     raise e
  unit-ceph-osd-4[27573]: 2016-03-14 11:36:20 INFO unit.ceph-osd/4.mon-relation-changed logger.go:40 subprocess.CalledProcessError: Command '['ceph-disk', 'prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']' returned non-zero exit status 1

  /dev/sdb is a normal rotary disk on this node. Partitioning went fine
  at :14 and :16, but at timestamp :20 mkfs complained that /dev/sdb1
  does not exist. Should we wait there perhaps?

  This is from https://ci.lscape.net/job/landscape-system-tests/1366/

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1557158/+subscriptions