[Bug 1522130] Re: Openstack API Charms have a naive haproxy connection timeout

[David Ames]

For the record *all* of the timeout values could use reconfigurability.
I wound up changing

    timeout queue 1000
    timeout connect 1000
    timeout client 30000
    timeout server 30000

to

    timeout queue 30000
    timeout connect 30000
    timeout client 90000
    timeout server 90000

This all in the haproxy.cfg template file
defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    timeout queue 1000
    timeout connect 1000
{% if haproxy_client_timeout -%}
    timeout client {{ haproxy_client_timeout }}
{% else -%}
    timeout client 30000
{% endif -%}

{% if haproxy_server_timeout -%}
    timeout server {{ haproxy_server_timeout }}
{% else -%}
    timeout server 30000
{% endif -%}

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1522130

Title:
  Openstack API Charms have a naive haproxy connection timeout

Status in cinder package in Juju Charms Collection:
  Confirmed
Status in glance package in Juju Charms Collection:
  Confirmed
Status in keystone package in Juju Charms Collection:
  Confirmed
Status in neutron-api package in Juju Charms Collection:
  Confirmed
Status in nova-cloud-controller package in Juju Charms Collection:
  Confirmed

Bug description:
  On OpenStack API charms haproxy sets timeout connection to 1 second.
  This is reasonable only in an ideal scenario. Connections to backends
  on a busy cloud may take much longer. Even haproxy docs suggest 4 to 5
  seconds [1].

  defaults
      log global
      mode tcp
      option tcplog
      option dontlognull
      retries 3
      timeout queue 1000
      timeout connect 1000

  This causes mysterious errors like the following:

  On neutron-api:
  ConnectionRefused: Unable to establish connection to http://10.245.161.159:9696/v2.0/security-groups.json

  On keystone:
  ERROR: HTTPConnectionPool(host='10.5.51.133', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'httplib.BadStatusLine'>: '')

  On nova-cloud-controller:
  requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.245.161.158', port=8774): Max retries exceeded with url: /v2/ba51369fa6444c6d9c2be7264675741f/servers/373ed532-9819-4f2b-b32a-4d27147d0338/os-interface (Caused by <class 'httplib.BadStatusLine'>: '')

  When max_retries is forced > 1 it becomes only slightly less opaque:

  urllib3.connectionpool - WARNING - Retrying (5 attempts remain) after
  connection broken by 'BadStatusLine("''",)':
  /v2/ba51369fa6444c6d9c2be7264675741f/servers/32eed1a0-6c9d-497f-b9fb-
  aac0fa748858/os-interface

  Set timeout connect (and timeout queue while we are at it) to sane
  defaults and optionally make this configurable.

  [1] https://cbonte.github.io/haproxy-
  dconv/configuration-1.5.html#4-timeout%20connect

To manage notifications about this bug go to:
https://bugs.launchpad.net/charms/+source/cinder/+bug/1522130/+subscriptions