[Bug 1636322] [NEW] upstart: ceph-all service starts before networks up

[Billy Olsen]

Public bug reported:

As reported in upstream bug http://tracker.ceph.com/issues/17689, the
ceph-all service starts at runlevels [2345] and introduces a race
condition which allows the ceph service (e.g. ceph-mon) to start prior
to the network the service binds to is up on the server. This causes the
service to fail on start because it was unable to bind to the specific
network the service is configured to listen on.

A work around is to provide a post-up directive to the network stanza
configuring the network device in the /etc/network/interfaces file which
restarts the necessary ceph service.

[Impact]

 * Ceph service fails to start on reboot of machine/container when
networking takes some time to come up.

 * The provided patch to the upstart service configuration adds the
static-network-up event as a dependency for the start on service
directive. The static-network-up event is started after all the network
stanzas have been processed in the necessary config files.

[Test Case]

* Configure multiple network interfaces and have the ceph service bind
to one of the last configured network devices to introduce a delayed
start of the network interface.

[Regression Potential]

* Upstream previously had the directive to start the service after any
network-device-up for a network which is not the loopback interface.
This caused some "weirdness" to be seen when the multiple network
interfaces were configured. This was likely due the events that it keyed
on being the local filesystems being available and a single network
interface being available. This would add the change to start only after
all the network interface stanzas are processed in the /e/n/i
configuration files.

* Additionally, this will cause some ceph services to start later than
they previously would have since this change causes additional start
dependencies. However, the results should be that the interfaces have
always had a chance to be started prior to the attempt to start the ceph
service.

** Affects: ceph (Ubuntu)
     Importance: Undecided
     Assignee: Billy Olsen (billy-olsen)
         Status: New

** Changed in: ceph (Ubuntu)
     Assignee: (unassigned) => Billy Olsen (billy-olsen)

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph in Ubuntu.
https://bugs.launchpad.net/bugs/1636322

Title:
  upstart: ceph-all service starts before networks up

Status in ceph package in Ubuntu:
  New

Bug description:
  As reported in upstream bug http://tracker.ceph.com/issues/17689, the
  ceph-all service starts at runlevels [2345] and introduces a race
  condition which allows the ceph service (e.g. ceph-mon) to start prior
  to the network the service binds to is up on the server. This causes
  the service to fail on start because it was unable to bind to the
  specific network the service is configured to listen on.

  A work around is to provide a post-up directive to the network stanza
  configuring the network device in the /etc/network/interfaces file
  which restarts the necessary ceph service.

  [Impact]

   * Ceph service fails to start on reboot of machine/container when
  networking takes some time to come up.

   * The provided patch to the upstart service configuration adds the
  static-network-up event as a dependency for the start on service
  directive. The static-network-up event is started after all the
  network stanzas have been processed in the necessary config files.

  [Test Case]

  * Configure multiple network interfaces and have the ceph service bind
  to one of the last configured network devices to introduce a delayed
  start of the network interface.

  [Regression Potential]

  * Upstream previously had the directive to start the service after any
  network-device-up for a network which is not the loopback interface.
  This caused some "weirdness" to be seen when the multiple network
  interfaces were configured. This was likely due the events that it
  keyed on being the local filesystems being available and a single
  network interface being available. This would add the change to start
  only after all the network interface stanzas are processed in the
  /e/n/i configuration files.

  * Additionally, this will cause some ceph services to start later than
  they previously would have since this change causes additional start
  dependencies. However, the results should be that the interfaces have
  always had a chance to be started prior to the attempt to start the
  ceph service.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1636322/+subscriptions