[Bug 1522130] Re: Openstack API Charms have a naive haproxy connection timeout

[David Ames]

** Description changed:

  On OpenStack API charms haproxy sets timeout connection to 1 second.
  This is reasonable only in an ideal scenario. Connections to backends on
  a busy cloud may take much longer. Even haproxy docs suggest 4 to 5
  seconds [1].
  
  defaults
-     log global
-     mode tcp
-     option tcplog
-     option dontlognull
-     retries 3
-     timeout queue 1000
-     timeout connect 1000
+     log global
+     mode tcp
+     option tcplog
+     option dontlognull
+     retries 3
+     timeout queue 1000
+     timeout connect 1000
  
- Set timeout connect (and timeout queue while we are at it) to sane
- defaults and optionally make this configurable.
+ This causes mysterious errors like the following:
+ 
+ On neutron-api:
+ ConnectionRefused: Unable to establish connection to http://10.245.161.159:9696/v2.0/security-groups.json
+ 
+ 
+ On keystone:
+ ERROR: HTTPConnectionPool(host='10.5.51.133', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'httplib.BadStatusLine'>: '')
+ 
+ On nova-cloud-controller
+ When max_retries is forced > 1 it becomes only slightly less opaque:
+ 
+ urllib3.connectionpool - WARNING - Retrying (5 attempts remain) after
+ connection broken by 'BadStatusLine("''",)':
+ /v2/ba51369fa6444c6d9c2be7264675741f/servers/32eed1a0-6c9d-497f-b9fb-
+ aac0fa748858/os-interface
+ 
+ 
+ Set timeout connect (and timeout queue while we are at it) to sane defaults and optionally make this configurable.
  
  [1] https://cbonte.github.io/haproxy-
  dconv/configuration-1.5.html#4-timeout%20connect

** Package changed: charms => nova-cloud-controller (Juju Charms
Collection)

** Also affects: neutron-api (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: keystone (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: glance (Juju Charms Collection)
   Importance: Undecided
       Status: New

** Also affects: cinder (Juju Charms Collection)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1522130

Title:
  Openstack API Charms have a naive haproxy connection timeout

Status in cinder package in Juju Charms Collection:
  New
Status in glance package in Juju Charms Collection:
  New
Status in keystone package in Juju Charms Collection:
  New
Status in neutron-api package in Juju Charms Collection:
  New
Status in nova-cloud-controller package in Juju Charms Collection:
  New

Bug description:
  On OpenStack API charms haproxy sets timeout connection to 1 second.
  This is reasonable only in an ideal scenario. Connections to backends
  on a busy cloud may take much longer. Even haproxy docs suggest 4 to 5
  seconds [1].

  defaults
      log global
      mode tcp
      option tcplog
      option dontlognull
      retries 3
      timeout queue 1000
      timeout connect 1000

  This causes mysterious errors like the following:

  On neutron-api:
  ConnectionRefused: Unable to establish connection to http://10.245.161.159:9696/v2.0/security-groups.json

  
  On keystone:
  ERROR: HTTPConnectionPool(host='10.5.51.133', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'httplib.BadStatusLine'>: '')

  On nova-cloud-controller
  When max_retries is forced > 1 it becomes only slightly less opaque:

  urllib3.connectionpool - WARNING - Retrying (5 attempts remain) after
  connection broken by 'BadStatusLine("''",)':
  /v2/ba51369fa6444c6d9c2be7264675741f/servers/32eed1a0-6c9d-497f-b9fb-
  aac0fa748858/os-interface

  
  Set timeout connect (and timeout queue while we are at it) to sane defaults and optionally make this configurable.

  [1] https://cbonte.github.io/haproxy-
  dconv/configuration-1.5.html#4-timeout%20connect

To manage notifications about this bug go to:
https://bugs.launchpad.net/charms/+source/cinder/+bug/1522130/+subscriptions