[Bug 1522130] Re: Openstack API Charms have a naive haproxy connection timeout

[Ryan Beisner]

Yes, I think a charm config option (and/or a more sane timeout default
value) would be great, for all os-charms which employ haproxy.

I've observed the "''" return during times of elevated load, in neutron-
api, nova and dashboard.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1522130

Title:
  Openstack API Charms have a naive haproxy connection timeout

Status in cinder package in Juju Charms Collection:
  New
Status in glance package in Juju Charms Collection:
  New
Status in keystone package in Juju Charms Collection:
  New
Status in neutron-api package in Juju Charms Collection:
  New
Status in nova-cloud-controller package in Juju Charms Collection:
  New

Bug description:
  On OpenStack API charms haproxy sets timeout connection to 1 second.
  This is reasonable only in an ideal scenario. Connections to backends
  on a busy cloud may take much longer. Even haproxy docs suggest 4 to 5
  seconds [1].

  defaults
      log global
      mode tcp
      option tcplog
      option dontlognull
      retries 3
      timeout queue 1000
      timeout connect 1000

  This causes mysterious errors like the following:

  On neutron-api:
  ConnectionRefused: Unable to establish connection to http://10.245.161.159:9696/v2.0/security-groups.json

  
  On keystone:
  ERROR: HTTPConnectionPool(host='10.5.51.133', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'httplib.BadStatusLine'>: '')

  On nova-cloud-controller
  When max_retries is forced > 1 it becomes only slightly less opaque:

  urllib3.connectionpool - WARNING - Retrying (5 attempts remain) after
  connection broken by 'BadStatusLine("''",)':
  /v2/ba51369fa6444c6d9c2be7264675741f/servers/32eed1a0-6c9d-497f-b9fb-
  aac0fa748858/os-interface

  
  Set timeout connect (and timeout queue while we are at it) to sane defaults and optionally make this configurable.

  [1] https://cbonte.github.io/haproxy-
  dconv/configuration-1.5.html#4-timeout%20connect

To manage notifications about this bug go to:
https://bugs.launchpad.net/charms/+source/cinder/+bug/1522130/+subscriptions