[Bug 1599636] Re: pause/resume failing on HA deployment: haproxy is running

[Ursula Junque]

** Description changed:

  I have an HA mitaka cloud, and am checking the pause/resume actions of
  its units.
  
- When trying to "action pause" the units of the services that use
- hacluster (e.g. keystone, cinder, neutron-api), the action fails *most
- of the time* because haproxy is running. "juju action pause" tries to
- stop haproxy service, but once pacemaker/corosync detect haproxy isn't
- running they restart it, the service is never really stopped and the
- action fails. Oddly enough, nova-compute-kvm works every time.
+ When trying to "action pause" the units of most services that use
+ hacluster (keystone, cinder, neutron-api, ceilometer, nova-cloud-
+ controller, ceph-radosgw and less often openstack-dashboard), the action
+ fails *most of the time* because haproxy is running. "juju action pause"
+ tries to stop haproxy service, but once pacemaker/corosync detects
+ haproxy isn't running service is restarted, service is never really
+ stopped and the action fails. Oddly enough, nova-compute-kvm works every
+ time.
  
  Keystone example:
  $ juju action fetch 783e4ee0-a498-42e3-8448-45ac26f6a847
  message: 'Couldn''t pause: Services should be paused but these services running: haproxy, these ports which should be closed, but are open: 5000, 35357, Paused. Use ''resume'' action to resume normal service.'
  status: failed
  
  juju logs excerpt for the paused unit:
  https://pastebin.canonical.com/160484/
  
  juju status of the whole environment:
  https://pastebin.canonical.com/160483/
  
  /var/log/syslog excerpt right after juju action pause is issued:
  https://pastebin.canonical.com/160379/
  
  nova-compute-kvm units consistenty pause successfully. All others
  sometimes work, but vast majority attempts fail. This could indicate
  that something incidental is been relied upon (e.g. assuming network is
  "fast" enough that races aren't an issue).

** Also affects: neutron-api (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: ceilometer (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: ceph-radosgw (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: nova-cloud-controller (Juju Charms Collection)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1599636

Title:
  pause/resume failing on HA deployment: haproxy is running

Status in ceilometer package in Juju Charms Collection:
  New
Status in ceph-radosgw package in Juju Charms Collection:
  New
Status in cinder package in Juju Charms Collection:
  New
Status in keystone package in Juju Charms Collection:
  New
Status in neutron-api package in Juju Charms Collection:
  New
Status in nova-cloud-controller package in Juju Charms Collection:
  New
Status in openstack-dashboard package in Juju Charms Collection:
  New

Bug description:
  I have an HA mitaka cloud, and am checking the pause/resume actions of
  its units.

  When trying to "action pause" the units of most services that use
  hacluster (keystone, cinder, neutron-api, ceilometer, nova-cloud-
  controller, ceph-radosgw and less often openstack-dashboard), the
  action fails *most of the time* because haproxy is running. "juju
  action pause" tries to stop haproxy service, but once
  pacemaker/corosync detects haproxy isn't running service is restarted,
  service is never really stopped and the action fails. Oddly enough,
  nova-compute-kvm works every time.

  Keystone example:
  $ juju action fetch 783e4ee0-a498-42e3-8448-45ac26f6a847
  message: 'Couldn''t pause: Services should be paused but these services running: haproxy, these ports which should be closed, but are open: 5000, 35357, Paused. Use ''resume'' action to resume normal service.'
  status: failed

  juju logs excerpt for the paused unit:
  https://pastebin.canonical.com/160484/

  juju status of the whole environment:
  https://pastebin.canonical.com/160483/

  /var/log/syslog excerpt right after juju action pause is issued:
  https://pastebin.canonical.com/160379/

  nova-compute-kvm units consistenty pause successfully. All others
  sometimes work, but vast majority attempts fail. This could indicate
  that something incidental is been relied upon (e.g. assuming network
  is "fast" enough that races aren't an issue).

To manage notifications about this bug go to:
https://bugs.launchpad.net/charms/+source/ceilometer/+bug/1599636/+subscriptions