[Bug 1522130] Re: Openstack API Charms have a naive haproxy connection timeout

[Ryan Beisner]

Observable a symptom when attempting to delete nova security groups as a
normal user just using the cloud:

nova secgroup-delete XYZ
ERROR: ('Connection aborted.', BadStatusLine("''",))

http://paste.ubuntu.com/13623685/

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1522130

Title:
  Openstack API Charms have a naive haproxy connection timeout

Status in cinder package in Juju Charms Collection:
  New
Status in glance package in Juju Charms Collection:
  New
Status in keystone package in Juju Charms Collection:
  New
Status in neutron-api package in Juju Charms Collection:
  New
Status in nova-cloud-controller package in Juju Charms Collection:
  New

Bug description:
  On OpenStack API charms haproxy sets timeout connection to 1 second.
  This is reasonable only in an ideal scenario. Connections to backends
  on a busy cloud may take much longer. Even haproxy docs suggest 4 to 5
  seconds [1].

  defaults
      log global
      mode tcp
      option tcplog
      option dontlognull
      retries 3
      timeout queue 1000
      timeout connect 1000

  This causes mysterious errors like the following:

  On neutron-api:
  ConnectionRefused: Unable to establish connection to http://10.245.161.159:9696/v2.0/security-groups.json

  On keystone:
  ERROR: HTTPConnectionPool(host='10.5.51.133', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'httplib.BadStatusLine'>: '')

  On nova-cloud-controller:
  requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.245.161.158', port=8774): Max retries exceeded with url: /v2/ba51369fa6444c6d9c2be7264675741f/servers/373ed532-9819-4f2b-b32a-4d27147d0338/os-interface (Caused by <class 'httplib.BadStatusLine'>: '')

  When max_retries is forced > 1 it becomes only slightly less opaque:

  urllib3.connectionpool - WARNING - Retrying (5 attempts remain) after
  connection broken by 'BadStatusLine("''",)':
  /v2/ba51369fa6444c6d9c2be7264675741f/servers/32eed1a0-6c9d-497f-b9fb-
  aac0fa748858/os-interface

  Set timeout connect (and timeout queue while we are at it) to sane
  defaults and optionally make this configurable.

  [1] https://cbonte.github.io/haproxy-
  dconv/configuration-1.5.html#4-timeout%20connect

To manage notifications about this bug go to:
https://bugs.launchpad.net/charms/+source/cinder/+bug/1522130/+subscriptions